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Chapter 1 
' * , t . V 

» Introduction 

t' 

Purpose of tl^e Repo rt 

In this report, we review the qualitative 'research" 
methodologies employed by CSE* sy3 valuation Use Project over 
the^last sit years. The report is neither a paean, to 
qualitative methods nor an attack upon then, it represents 
a self-exaairfation of our best efforts to apply qualitative 
techniques to an important, and reasonably coaplex, 
educational .research problem: the study of evaluation 
information use in local schools. On the whole, all of 
these qualitative methodologies have**proven useful, 
although they have not been without difficulties and / 
limitations, in the report, the/ research methods are 
described and critiqued. v ~" 

.There are dual reasons for such a review. First, the 
time seems Tight for a methodological reassessment. 
Enthusiasm for qualitative methods burgeoned in the late- 
1970 «s and has ebbed a bit, since then, sone researchers 
have taken qualitative methods to -task based upon 
discouraging experiences (e.g. , Hiles, 1979).' The CSE 
experience has been less discouraging, although not without 
its lessons, it seems appropriate, and potentially' helpful 
to others, to share these experiences. 

.Second, the CSE Evaluation' Use Project has had the 

1 



opportunity to apply several qualitative research 

strategies* The Project's use of interview-based case 

studies, participant observation,, and seai-structured 

interview surveys constitutes an uncoaaonly broad 

# 

experiential base, which_others may find useful to exaaine. 
The Experience Base * 

Since i975, th^e CSE Evaluation ose Prpject has 
completed three aajor studies of locar schools 1 use of 
evaluation inforaation. The first study ~w actually a set 
of similarly conducted qualitative case studies — exaained 
evaluation in five ESEA Tit ^e I (coapensatory "education) cr 
Title IV-C (innovative) prograas. Bepeated^ open-ended, 
interviews were conducted with the evaluators and with 
adainistrators and staff of the five prograas, and an 
evaluation history* of 4 each prograa was developed. \ In " 
addition, the interviewees were asked to explain the uses 
aade of the evaluation findings. A naafr^tive rep<?rt on 

each prograa (i.e. "case") was critiqued by the 

I 

interviewees andother reviewers. After all five' case 
studied were completed, a cross-case analysis "was prepared 
(Alkin'et al., 1979). 

, Thereafter, twd further studies were conducted, this 

i * 

time within a single urban Califorpia school district 
iHetro Unified School District). One, the Evaluator Field 
Study (Daillak, 198of, used participant observation and 
interviews to trace the work of three on-staff evaluators 



within Metre), over t*he 1979-8 0 school year, the 
evaluators 1 work was observed, and institutional influenced 
upon their work were investigated. 1 

The other study, the User Intefview Survey (ilkin et 
•al. , 1980), adainistered a loosely- structured one-hour 
interview to sixty- five school evaluation "consuaers" 
(principals and thair subordinate administrative staff) 
■ within twenty-tjro schools* The intetviewees were 
questioned about recent progpaa planning or decisionmaking 
activities, and about their attitudes towards evaluation. 
Written suaaaries ol; the (tape-recorded), interviews were 
prepared by the interviewer and by ail independent 
tape-listener. These suaaaries then were use<^ to develop 1 
several analytical papers. 

The study aethodologies are reviewed in ^three chapters 
to follow. A brief epilogue suaaarizes our experience with 

7 1 

the aethods. 

Hie Problem Evaluation On der utilization 

There is substantial evidence that progf aa evaluation 
findings .-are seldoa use-d^Sy local school decisionaakers, 
such^^^^^g^E^staf f , prograa directors, or building s 
prin^^^^^mv : tA, 1978; Kennedy et al. , 1980), despite the 
fact t^^^^tuation activity is reguired of many federal * 

g * 

or state' categorical aid prograas. Local school districts 
generally coaply with the letter of jjhe evaluation 
requireaents iaposed upon thea. They conduct the student 



achievement testing required by funders, and they annually 

assess each ^prbgraa*s success in meeting its prespecified 

objectives. But schools do not seem to attenj} to or use 

the evaluation information they generate, at least not 

significantly. u 

Research has attempted to document the precise extent 

of local evaluation .use, to identify explanations for use 

— v — " 

or disuse, afti^ta. determine ways to increase evaluation 

use* Soae researchers have tried to improve evaluation. 

research methods in order to aake evaluation data 

objectively aore 'accurate and, they hope, aore useful* 

Others have exaained the interaction between evaluation and 

client systems, believing that evaluation use is decided 

aore by the decisionmakers 1 attitudes and intentions than 

bj objective properties of the information itself* 

Out research has emphasized the latter approach, 

focusing primarily on social, political, and organizational 

explanations for evaluation^utilization. To assess coaplex 

vsocio-organizational interactions, we have relied heavily 

on qualitative studies of evaluation settings rather than 

employing more reduction is tic approaches, such as 

quantitative or siaulation-based research* ' 



Chapter 2 
^ Evaluation Case Studies 



RESEARCH DESCRIPTIOH 
In 1975 # when prior research on evaluation use was 

i 

* r 

scant, it seemed important to exaaine evaluations, in an 

exp locat ory w ay, i n-or der to discoTer the fac tors that 

miglit account for evaluation information use (or disuse) by 

local school decision makers* if a few plausibly important 

factors were to emerge from these initial studies', then 

they -might be investigated in later research* For example, 

if evaluation report formating appeared to affect 

information use, then la*ter research might exaaine the 

merits of alternative 'procedures for reporting an 

evaluation's findings to decision makers/ r 

v Choosing t]je. Case S.£udx Method 

.It-seemea logical to favor either- survey or field 

research set hods for the initial, exploratory studies. 

Neither field experiaents nor laboratory siaulations ~* 

appeared appropriate. Field experiments seemed *premature« 

P-aper- and- pencil simulations could have beeti instituted 

more quickly, at lowermost. Program circumstances, 

1 evaluation findings, and' reports all could be simulated. 

f * 

- By introducing systematic variations lnto\he simulated 
evaluations, one could explore the effects of proposed new 
evaluation methods. However, we hesitated to try building 
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a simulatron for a phenomenon about which so little was 
'known* J s 

Surveys and exploratory field studies remained as 
plausible research strategies — and both had their 
advantages* Questionnaire and interview surveys could < 
re^ch larger numbers of respondents, providing a more " 
representative view of evaluation. Exploratory field 
studies — case studies, in-depth intervie wing, participant 
observation, and related methods — could provide nore/' 
detailed research data, however* *> 

For our purposes when the initial" research was 

planned, case studies based on detailed interviews seemed 

Ko be the most desirable approach.' The argument for ■ 

exploratory field studies'has been made often before: 
N , • 

Sociologists usually use •••[.field research ]* 

when they are especially interested in 

understanding a particular organization or 

substantive problem rather than demonstrating 

relations between abstractly defined variables. 

They attempt to pake their research theoretically 

meaningful, but they assume that; Ahey do not 

know ^nougfi about the -organization prior i, 

to identify relevant, problems ana- hypotheses ana 

that they must discover these iatKe course 

of the research (Becker, 1958, 652-653)' < „ 

Investigating evaluation situations through 

naturalistic field research thu^ had much to offer, but the 

problem of deciding just what "shape the field research 

effort should take reaained to be decided, should there be 

participant observation, or would interviews be the better 

approach? How should the sites be found, how long would we 



spend yith then*, and what, if anything, should we look for? 
These questions were resolved as described below* 

C ase stud.y Proced utes 
hs is topical in' qualitative field research, exact 
procedural details varied from case, to case. Nevertheless, 
the major proceiinfaj steps and their sequence 'were quite 
similar across the cases* * , 

Overview * 

> . * ^ 

The case 'studies were primarily interview-based and 

* * 

retrospective (i„e. , focusincfon a previously completed • 
- evaluation^ . . Several in-deptwjntervie ws were conducted 
with the .operational^taff and .the evaluator of the program 
selected for stJuSy. To* supplement the interviews, 
documentary evidence, such as program proposals, evaluation 
rep&rts, 5 andyso forth, were reviewed. * The research sought 

"•S 

to discover the way that ah evaluation, ana the findings .it 
procldced, had fit into the program* s total operations. A 
primary concern was to discover whether an evaluation ^ad 
influenced .program activities and why i't had the influence 
{or lack of influence) it did. * mJ 

In several cases, one or more year-long-* evaluations 
had-s^een completed and a follow-on evaluation was in 
progress. In these ca^es, the interviewer constructed an 
account of the completed* evaluations in the usual fashion 
(described below) and also update^ this account at 
intervals to reflect contemporaneous developments* 



The final product of £ach case study v^s' a narrative case - 
report summarizing the study findings. Program context was 
described; one or more evaluations were recounted; 
evaluation influence, 'if any, was described; and the case 
was briefly analyzed, focusing on plausible explanations 
for the observed degree of evaluation influence. 
After all five case studies were completed and reported (a 
aatter of two to three years* work, since the studies were 
begun at staggered intervals), an integrative analysis was 
prepared. This reanalysis isolated^ nuaber of coanon 
factors that is, ones recurring across several cases — 
#&ich* seeaed to encourage, constrain, or in soae other way 
/^.affect evaluation information use in the cases studied.* 
Similar factors were clustered until a final framework of 
factors influencing evaluation' utilization , was produced. 
The case study'narratives and the framework were published 
together as the concluding product .in the initial research 
phase "(see Alkin et al. , 197?) . 
Procedures for th_e in dividu al Case studies 

Site Selection , 'only local school /prograas 
receiving ESEA Title I (compensatory education) or Title 
X?-C (innovative program) funds were selected. The choice ' 
of local ESBA Title 1 and iv-C programs reduced \he * > 
.potential program diversity' somewhat, yet still provided an 
abundant . progr a a pool from which to choose sites. In 
addition,' ESS A evaluations (required as a condition of 



m9 > ' 

funding) ■ account for much' evaluation activity '* in . local 

educational agencies* J * 

* • 

- Because only a saall number of studies were tp be 
ucted (five in all, selected^ one at a time over two 
r years), elaborate effotts at randomized sampling were aot 
undertaken* - Instead sites were identified either through f 
contacts .in the public schools or from among the *' 
respondents to an earlier mailed survey which th^ 1 
EvaluatidS Use Project had conducted in California school 
'districts. The major c^riteria^ for site selection were: 
(a) the existence of a Title I or IV-e program in at least 
its second year of operation r [by geographic proximity, fc) 
\ willingness of site personnel to participate, and (d) a 
subjective assessment of the Program^ suitability for the^/^ 
stttdy. ♦ " 

A program which ^.et the first criterion would 
necessarily have gone through at least one fullyannual 
evaluation cycle; thus,' the interviews could 
•retrospectively trace the impact of the completed ' 
evaluation (s) and also examine any ongoing evaluation 
activities. The Second criterion, geographic proximity, 
meant simply that the program had to be within reasonable 
traveling distance of\>ur Los Angeles base — effectively, 
"within the Southern California area* The host*s * 
willingness to participate voluntarily was a third, crucial 
criterion. He had no power to force ourselv^sT upon any 
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school site* Potential hosts Kad\o be persuaded to 
/ cooperate with the research. As a consegueApe*, some' amount 
of self-selection was inevitable" in the case study. 

The final criterion, a subjective judgment of 
"suitability," was-* complex* \s the cases were selected, an 
effort was made to diversif v^t>e sample within the broad 
parameters set by the other criteria* ie tried to select 
programs of varying types: two ; were Title I, three were 
Title IV-C; three herein large, urban districts, two in 
smaller, suburban districts; * two cases involved inter*i^f 
evaluators who worked full time within evaluation units of 
their school districts; in two. other instances the internal 
evaluators were district employees who were not part of an 
. organized evalultion unit; and in the last case, the 

evaluator was an outride consultant* Besides striving for 
diversity, we sopght to avoid programs which might be 
misleadingly atypical — for example, a districts showcase 
effort, treated markedly, differently from its other 
federally funded programs. 

Initial contac ts with 5itq Personnel, . First y 
discussion^ typically were with the selected program f s 
evaluate {In only one case was the program director 
contacted first). Typically/ the first contact was a phone 
c^S^ersation in which the proposed study's purposes and 
Methods were outlined.' In each case, ' the person contacted 
agreed to further discussions, usually through cne or nore 
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personal meetings but also occasionally involving an i 
.exchange of letters. TJiese initial discussions quickly 
drew in the .program director, £he evaluator, and pefhaps a 
school official (such as a principal or district j 

administrator)*, as well as est. project members (typicaily, 

t \ 

the Project Director and £he interviewer). 

The initial personal meetings introduced the research 
'to prograa administrators, ffe tried "to project a sincere/ 
njon judgmental interest *'ln their program and" its evaluation 
Anonymity was promised to the participants; in^ray 
research' reports, pseaaonyis would, be substituted for the 
names of cities, districts, schools, programs, and 
individuals, and other identifying elements jwould be 
disguised in ways which would preserve anonymity without! 
significantly distorting the; facts, - 

Hone of the sites initially contacted decli^ad' to 
participate. Several factors probably contribute^*:© this 
high participation rate. Selecting sites through V- 
professional acquaintances' certainly played a key, role, as 
did .the reputation of the senior researcher and the 
agencies. 

Data Collectio n. Hone of the meetings wi'th % subjects 
were tap*;- recorded. Written notes were taken, and the 
interviewer expanded and elabbrated these notes after/ 
leaving the interview situation. An effort was made to 
keep the interviews; -flexible in format ^md informal in 
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tone* ^Initial interviews we're particularly wide-ranging* 
I Later, as the interviewer became more faailiar with the 
case details, the interviews often focused on questions 
prepared in. advance* by the interviewer* 

In the first meeting, the interviewer requested copies 
of aajor program docuaents, including funding proposals, 
program progress reports, reports of previously completed 
evaluations, and any other documents thp program staff felt 
would he informative. The documents provided an "official" 
description of the -program and its evaluation, g*nd they 
were used as background for subsequent inter vie ws # 

After the introductory- aeeting (s) , a round of private 
interviews was conducted with the *site program director, 
t$ program evaluator, the princip^, and any others 
. actively involve'* in prdgraa decision making. In "these 
interviews , the informants were encouraged to describe the 
program and 1 its evaluation in their own words. 0§ually„ 
the interviewer simply indicated our broad interests — the 

! 

* > v 

history, context, and events of the program and its 
evaluation, t and the evaluation's influence upon the program 
~ and then the informant guid.ed much of the* conversation, 
with the interviewer asking clarifying questions. 

The evaluation literature did suggest that certain 
factors would influence evaluation use: the background and 
training of the individuals involved; the evaluation's 
purposes and procedures; personal' interactions daring the 

12 
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evaluation; and the manner in ^which evaluation findings 
were 'communicaited. If these f|ctors were not covered in * 
the informant* s spontaneous remarks, the interviewer raised 
them later in the interyfof. 

Ho single interview could fully cover all the^topics 
of interest* Each interview had to* be limited to an hour 
or two in length, and, 'besides, the need for clarif ica tion 
^>ften became apparent only later, after interviewing other 
informants, fherefore, 'key informants wre usually 
; interviewed several times. The multiple interview sessions 
appeared to increase the rapport between the interviewer 

and the informants. 

v 

. Interviews with the evaluate ^and the program director 
• frequently indicated potential new interviewees — for 
example, a particularly influential teacher, principal, or 
counselor. These new informants, Mh^n interviewed, 
sometimes suggested contacting others. Thus the interviews 
"snow- balled". 

# 

Case , analysis . The multiple interviews provided a 
fairly detailed description of each case. The informants • 
differing vantage points generally complemented one, 
another, each filling in a part of the total picture of 
evaluation in the program being studied* ' Event 
descriptions could be "triangulated" (Guba, 1978) by 
conparing different accounts of , the sane actions or : events. 
Conflicting descriptions were not a cobboq problem j^he fe« 

* c 
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apparent conflicts were resolved through follovup contacts 
with those involved. 

Once the set of interview data, appeared complete — as 
evidenced, for example, By considerable redundancy in the 
data received — the interviewer prepared a first draft 
case report* Several steps were then followed to refine 
ythis draft and insure the final report's accuracy. 

First, the CSB Project Director reviewed the event 
descriptions for consistency, completeness, and 
plausibility. Frequently, this review raised questions 
that the existing data could not resolve. To provide 
answers, further personal interviews or telephone 
discussions were held with the informants, and t^-draft 
report was revised. This cycle of review, supplementary 
data collection, and revision was repeated until a draft 
satisfactory to the CSE Project was" developed. 

The case report draft, was next circulated to tHe 
• interviewees .themselves for c&mment. a report cjtfpy was 
given to each key informant (e valuator, school projects; 
director, etc) , and a personal interview was conduct^ 
after allowing the informant one or two weeks -to review the 
report. 

The informants were extremely' helpful reviewers^ They 
identified factual errors and suggested alternate data 
interpretations." trhete necessary, additional data were 
collected to thrash out controversial points. Then, based 
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upon all the information available, the case report was 

edited into final fora. The fina JL report conformed to each 

informant's perspective on some points and differed on 

others. All in all, it represented the Projects best ^ 

effort at a balanced and accurate case presentation* 

♦ 

As a final effort at fairness, and to give readers a 
better sense of the study«s accuracy, the key informants 
then were asked to respond to an open-ended questionnaire 
assessing the final case study report and general issues 
pertaining to evaluation utilization* Their responses were 
excerptea and lightly edited (with their approval) to 
.produce narrative aeaos, appended to each case report* 

Thus, the last word was given to the major characters in 

1 

each case* 

*■* 

Comparative Analysi s o£ the Case Stodies . 
By five ca-ses had*%een individually analyzed, it 

seeaed desirable to review all the cases in order to 
identify coaaon forces underlying evaluation use. 

The comparative analysis began with a review of the 
five case v reports and an attempt to reduce fh^ir detailed* 
narrative data more manageable capsule summaries. 
Project staff members ip*aependently developed a set of 
"utilization concept-cards" for each case. That is, a 
separate 3x5 card was prepared fo£_each . critical context 
factor, events situation, or participant in each case. 
Criticality was judged subjectively, based on the 



importance that the factot or event seemed to have in 
determining the way the evaluation was performed and the 
way its findings were used. 

Next / 'the Project teas set to aiscuss each case, 
comparing and merging the concept-cards 'they had developed. 
The merged cards were then arranged schematically on a 
bulletin board to illustrate the flo* of events and 
influences that seemed to characterize each s case. Finally, 
the scheoas for the "five causes were compared in a search 
for analogous elements or patterns* 

Belated elements did appear across the cases. For • 

9 

example, in most of the cases there were card entries which 
related to evaluator credibility. Together, the Project 
staff discussed, grouped, and labeled the card groupings 
until, ultimately, about ten broad categories emerged. The 
categories formed the first version of a framework for 
analyzing evaluation situations. The category labels 
captured the terms useful for explaining why the 
evaluations studied had been influential or had had little 
effect* , N 

Shifting from the concept-cards to a new format of 
taped group discussions, the Project- members refined the 
tentative framework, in the process reconfiguring seme of 

a 
i 

the original categories. These iittensive discussions 
occupied several weeks. Gradually, the analysis 
stabilized, first at the category level and then, at the 
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level of coaponent eleaents (termed "properties" , following 
Glaser S Strauss, 1965). The final Version of the. ** 
fraaework is given in Appendix &^ . 

A paraaonnt concern was. to link the fraaework tightly 
to the cases to avoid imposing any preconceptions on the 
^data. fiuch of the group discussion centered on whether the 
data supported the categories as then stated, and whether 
sone other formulation would conform more closely to ths- 
case findings* 
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Review of the Casg Stii gy Purposes 
The research's initial assuaption* was that evaluation 
was, not very influential* This assuapt ion reflected 
prevailing opinion, as expressed in a*ny published. 
complaints about evaluation underutilization (<*«g. , Guba, 
1969; Hann, 1972). with the Evaluation Case Studies, the 
Project hoped to explain why school prograa evaluations had 
influenced prograa planning and decisioria^king so 

i 

ainiaally. In . the process, it hoped to discover ways that 
evaluators and prograa aanagers aight increase evaluation's 
usefulness. 

Canvassing the literature, one could coapile a long 
list of potential causes of evaluation underutilization, 
soae* organizational, soae procedural, and soae } 
interpersonal (c.f. Patton et al. , 1975). But there was 
little evidence underpinning this potpourri of causes, 
because, prior to 1975, evaluation influence and use had 
received little studj* 

The case studies -dere to explore school settings in*, 
order to discover which , if any, of the hypothesized causes 
of evaluation disuse appeared significant - , to identify any 
other iaportant causal factors, to docuaent evaluation 
strategies used by practitioners, and to deteraine how 
extensively evaluation inforaation was used in several 

18 
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programs, it ^seemed likely that case studies, producing, 

detailed descriptions of a few school evaluations, would 

further the#e ends economically and expeditiously. It w^s 

expected that subsequent studies — for example, 

experiments employing in~basket written simulations 

would refine the case studies 1 conclusions. The case 

studies 1 detailed descriptive data could, in fact, be used 

to develop more realistic simulations. The case studies 1 

exploratory nature, ^underlined by the expectations for 

followup wprk, partially relieved our apprehensions about 

employing qualitative methods, which, were then rather 

uncommon in educational studies. 

ffes ^arcfr Design Decisions 

Several .design decisions shaped the research, one was 

the fundamental decision to employ case, studies rather than 

some other research strategy.. Thereafter, a decision was 

✓ . • v 

made to examine already completed evaluations, i.e#, to 

conduct retrospective studies, rather than to follow the 

unfolding events of a current evaluation* Finally, the 

decision was made to collect primarily qualitative data 

(mainly, narrative accounts of events^ through intensive, 

open-ended interviews with a few local evaluation 

participants, 

' Case studies wera not the only research option 

considered, 4 Simulation studies employing written 

situation portrrayals, mock reports and memoranda , etc. — 

* 
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w6re- considered as a aeans of ^perimpntally investigating 
the factors influencing evaluation use. Based, upon advice- 
and our own judgment, simulation studies were deferred, 
first temporarily and then indefinitely. Initially, 'not 
enoagh was known about school program evaluations to he 
able to develop simulations in which one could place 
confidence. Later, after the case studies were completed, 
it appeared that interactions aaong evaluators and clients 

.... V 

were crucial to utilization, and, because these 
interactions resisted laboratory sinulation, plans for such 
studies were set aside. 

Questionnaire surveys were also considered. Indeed, a 
saall, pilot survey was administered to aeahers of a' 
California evaluation society. -But evaluation influence 
and information use seemed to elude measurement in the . 
questionnaires, and few interesting results emerged. An 
initial program of case study research seemed th"e most 
desirable option. The case studies' would permit 
conversations with evaluation participants, allowing for 
intensive questioning about evaluation information use and 
its contributing causes* Insights developed in the case 
studies could target later research toward the aost 
productive issues. , 

Practical and theoretical considerations coabined to 
yield a retrospective -case study approach. Inforaation 
use could be assessed, fully only at the conclusion of an 

( . 
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evaluation, after all the evaluation findings had been * k 

i ' 
reported. The literature hinted, in fact, that an ' 

•i 

evaluations full impact aight not be. -felt until several 

S ' ^ \* , 
months a«fter the evaluation, report 1 s- release (Davis S V 

L / ' 

Salasin, 1975). Consequently, a contemporaneous study of 

.an evaluation, from its / inception until - the final .stages of 

■ ( 

utilization, could^take substantially more than a year to 
complete, a retrospective study, however, could begin 
several months afmPthe evaluation's completion, when 
utilization mi§ht be fully Apparent, and quickly construct 
X history of past e vents* > ' The retrospective approach 

praised* faster results, at Hover total cost* These 

. — - ** « 

advantages to the retrospective strategy carried the day, 
even though it was clear that retrospective §ata would not 
be as complete as that available >in a contemporaneous 
study* 

Onde a retrospective approach was chosen, aany other 
details fell into place* " Observations were ruled out, „ 
except as current observations light shed light on past 
events. ' Documentary evidence would be examined,, but could . 
be expected to be rather limited. Partie^^^s 1 fc 
recollections were the major resource from wh£ch to develop 
the case data* Although recollections could be, tapped 
through questionnaires or f ixed^rotocol interviews, * > 
informal open-en.ded interviews sefcied to be the most ' 
expeditious means to detailed case, descriptions. Horeover, 

\ • 
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i . S f°^ i nterv is*s 'let the interviewees use their own words 

and their own organizing theaes, jrather than forcing then, 
to fit thdir descriptions to a prfideterained fraaew^rk. in 
an inforaal interview, interviewees could introduce issues 
which otherwise might have eluded attention* and 
. interviews could fully explore subtle uses of evaluation 
, ' inforaation, such as wjxen evaluation strengthened a 

decisionmaker f s opinion about a progtfaa. 

The research began with an assumption that evaluators 
and decisionmakers were keuy participants in evaluation, and 
-with the expectation that their actions largely would 
determine an evaluations ef fectiveness. In each case , 
c therefore, the evaluator and the prinfcipal local-prograa 
* decisionmaker were interviewed. One or .two influential 
"* .others (usually administrative subordinates to the primary 
decisionmaker) . were also interviewed if they appeared to 
• .influence the evaluation's design or the use aade of, the 
evaluation -results. ',«-"• 

p 1 - Bethodolog4?:^A Assessment , • 

Descriptive Detail w * 

t , A descriptive report, ranging froa forty to eighty 
^^ble<-sp^ced typewritten pages in length, was prepared fof 
each case. The reports followed, a coaaon pattern. 
* -Coaaunity and schorol context" were briefly descri&e^. The 

special program Receiving evaluatio^was described next, in 
^ somewhat greater detail* Attention waa given not just to 
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describing the program^ jaature; program history was also 
reported, 'including a description of the prog ram *s origins 
and rationale* - 

Hext, the program*s evaluation context was briefly 
described, including the program's evaluation requirements 
and the evaluatoc's perceptions of his or her role 

* • 

vis-a-vis the program* A more detailed description of the 
actual evaluation work, narrated, chronologically but'' 
without precise dates, followed* This included reportage 
of meetings between the evaiuator and school staff, 
evaluation procedures, and evaluation findings* The case 
reports concluded with a description of an/ uses made o$ 
the ^evaluation findings, and a synthesis of the 
participant explanations for use (or lack of use) of^ach' 

* e 

major evaluation finding* . 

The case reports were written primarily tm 
third-person narrative form* Quotations were few, because 
nope of the interviews were -.tape recorded and the 
interviewer had been, unable to enter many verbatim 
quotations in the handwritten notes* Documents, primarily 
evaluation reports, had been examined, and portions of 

c * 

their contents (e#g # , e,valuatipn descriptions and fc test 
results) were integrated into the narrative "case* reports. 
However, documents were not directly reproduced in the 
reports. The scarcity of direct gnotatiops detracted 
soaewhat froa the vividness of the case t re ports. 

4 
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Additional quotations or document excerpts would have 
provided more visible supporting documentation* 

^Obscured by the third-person narrative format was the 
fact that there were only thr^e or four primary 
interviewees for each case. Given the small number of 
interviewees^ ^fte-^ase ^tudies^ tietail uas high, an . 
indication of the intensity of the interviews* Hissing ' 
from the studies, however, was the.Jbreadth of coverage" that 
is possible when many persons are interviewed. For 
example, detailed data was not available on classroom 

-teachers 1 perspectives on evaluation tittle data was 
collected on the school environment, apart from that which 
seemed directly relevant to evaluation work. District-wide 
attitudes and perceptions were not explored in any detail, 
and th^re was little to place the studied evaluation in its 
full district context. 
Descriptive Accurac y 

There was strong reason to believe that the case * 

♦studies descriptions of evaluation "events were^JEastually 
accurate. In large part, this is because the case reports 
concentrated on events in which the interviewees had 

* 

directly participated, of^en jointly. - The study aethods — 
repeated interviews, probing for details, cross-checking 
facts with the other interviewees-, and submitting the draft 
report to the interviewees for review — all worked to 
verify' the factual descriptions of evaluation events. 
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Descriptions of evalution use .were aore difficult to 
verify. How can one be certain-thai evaluation inf oraation ' 
— or any other information — was "considered* by a 
decisionmaker? It is possible that the prograa staff who 
were interviewed exaggerated evalution use in order to 
respond to^jur obvious interest in instances of use* One 
safeguard, however, was that tht^ e valuators gave their own 
views on utilization^ which "stood as a partial check on the 
prograa staff.'tf assertions. 

Explanations for events were provided in- abundant 
detail in response to our persistent probing in the 
interviews, such explanations are inherently difficult to« 
verify. The interviewer was partially able to judge an 
explanations adequacy based upon its consistency with, 
l>ther data ana upon an assessment of the inte^v/ewee* 1 s 
candor and perceptiveness* Alsd> explanations were 
integrated into the case report, particularly in the final 
section on evaluation use, and we re ^available, therefore, 
for inspection by the other interviewees and by external 
reviewers. These reviews were a check on the 'explanations 1 
plausibility, aflthou^K^they were not proof of their 
accuracy, often, the explanations *ere of events* which had 
occurred weeks or aoriths prior to the interviews, adding 
another eleaent of uncertainty. 

Unquestionably, 'there were threats to the case 
studies 1 accuracy, as the preceding^ discussion indicates. 
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Bhat'is difficult to' convey, however, is the extent to 
.which these concerns were assuaged daring the research. 
The interviewees we/e generally very cooperative, in the 
course of the research, a positive relationship between 
interviewer and subjects usually developed, and there were 

* 

strong indications that the interviewees 1 remarks wer* 
candid and sincere. These indications, vivid to the 
interviewer but elusive to" describe/ gave us confidence 
that the case "studies accurately described evaluation* as it 
appeared to the interviewees, of lore^concern was whether 
the interviewees 1 understandings were adequate to explain 
•events. 4 
Nature of the Cross -Case Analysis . 

The tfross-case analysis categorized the individual 
studies* explanations for evaluation information use (or 
disusre). Each individual case study, in turn, had grounded 
its analysis of evaluation utilization in the explanations 
for use or disuse provided by the interviewees. The 
cross-case analysis generalized from the interviewees' 
explanations, by identifying similar explanations across 
the cases and labeling the generic factors or processes 
"which seemed to be represented. And, as a result, the 
cross-case analysis did not introduce entirely new 
ana 1 yt ic air concepts 

The categorization (or analytical framework, as it was 
labeled in Alkin et al., 1979) did not furnish a* predictive 

26 
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theory of evaluation use. That is, the categorization did 

not specify how. the identified factors or processes would 

affect evaluation information use, either singly *or in 

4 _ , interaction. Instead, a factor»s presence in the 

categorization simply indicated the factor*s potential 

significance. For example, the interrelationship between 

school site and school district was listed as a factor 

which could affect evaluation use, but the probable effects 

•of specific types of site-diWrict relationships were not 

) 

indicated. 

The cross-case, analysis 1 s mast significant 
contribution was that it identified, based upon empirical 
data rather than speculation, a limited set of influences 
upon evaluation use. The analysis was one step toward a 
more complete theory of evaluation utilization, with more ' 

. thorough investigation of the. identified factors being the 

• logical next step. 

Project discussions of the case data went further than 

) ' 

the published analysis in identifying the most important 
influences upon utilization. Our private conclusion was 
that the evalaator 1 s -and the school client f s casaitaent to' 
Baking evaluation useful was vitally important, it 
appeared, too, that evaluation information was most likely 
to be used when the evaluator had established a 
• consultative relationship with a local . school 
dec-^i on maker. Relationship's in which the evaluator acted 

! 
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as an auditor, aonitor, or detached critic appeared less 
successful. Through close consultations, an evaluation 
could b§ designed that would address actual decisionaaker 
interests, which could be guite(^3ifferent fro a the 
interests that night be inferred froa official program 
plans, and in consultations throughout the evaluation, the 
evaluator could involve the local client in decisions • 
affecting evaluation design, data*analysis, and 
interpretation. The result appeared to be aore aeaningful 
evaluation inforaation, in which the local client had 
^teveloped a sense of ownership. 

, It appeared that auch evaluation disuse was caused by 
the absence of this consultative evaluation approach, 
either because evaluators had chosen other roles or because 
policies or other external constraints interfered. The 
aost inf luentialyevaluators studied had evidenced this 
adaptive , ^consultative style rather than working solely 
froa f oraal prograa descriptions and routine evaluation 
procedures. t 

In suaaary, the published cross-case analysis 
categorized factors affecting evaluation utilization* 
Because these factors were drawn froa the individual case 
analyses^ the cross-case analysis had high face validity 

for readers who had found the individual case studies to be 

■> 

plausible, our private analyses had gone a step further. * 
We assigned priaacy to a subset at the categorized factots. 



namely th'e initiative shown by e valuators and 
administrators and the Use of a consultative evaluation 
approach. These conclusions were supporteoDy regularities 
in the case data: instances of information use appeared to 
be associated with special individual effort and close 
consultation between evaluator and local decisionmaker. 
Generalizabilitv o£ tt^ Findings , 

A traditional defense of the studies* generalizability 
is difficult* Personal contacts frequently were used to 
locate potential case sites, and subjects* participation 
was entirely voluntary, hence vulnerable to the biasing 
effects of^ self-selection. Although ther£ was an effort to 
introduce diversity into the sample by selecting different 
types of projects and various school district environments, 
greater efforts at scientific sampling could have been 
made. let, even if sites had been randomly selected, five 
cases would constitute a minuscule statistical sample from 
which to generalize findings. 

These difficulties are common to case study research, 
which- frequently relies upon sjiall samples, with sites 
selected, in part, for their convenience and cooperative 
response (Bogdan & Taylor/ 1975). The fact is that case 
studies 1 geperalizability seldom rests cn the same grounds 
as that oir»guantit,ative* surveys or experiments, if for no 
other* reason than small sample size. But case study ■ 
findings can apply to other situations, as reader 



response to such studies attests. To the- extent that 
others ao find the research aeaningful and Relevant, then 
there is evidence of generalizability. 

The publishea report of the case' study research (Alkin 
et al. , 1979) has received favorable reviews for its 
realisa and for the insights the studies provide. into 
evaluation practice* These reviews are perhaps the best 
eviaence that the findings are generalizafale beyona the 
five cases studied. 
Points of concern 

In hindsight, there were aspects to the case study 
aethods which aerit soae discussion, either because they 
represent practices we aight change, based upon today f s 
knowledge, or because they had raaifica tions which sight be 
overlooked, 

- Avoidance of Xa££ Recording . The interviews were ■ 
not tape recorded, primarily to avoid inhibiting, the 
interviewees. Although the Methodological literature - is 
divided on the issue of recording, recent Project 
experience," in the Oser Interview Survey* suggests that 
taping need not interfere with data guality. in the Use'r 
Interview Survey, interviewees adjusted quickly to the tape 
recorder, and they appeared to speak -candidly. 'we feel," 
now, that the case study interviewees would have adjusted 
siailarly* } 

Nevertheless, tape recordings are not 'an unmixed . - 
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blessing, even if interviewees are unintiaidated by the 
process* Tape handling, transcription, archiving, and 
transcript renews quickly create a sizable logistic 
burden. Had the ca-se study interviews been taped, however, 
ouch more detailed data would have been available froa 
which to prepare the case reports. On balance, we wouia 
tape if we had the decision to make today, but we would 
expect a aore time-consuaing and expensive study* 

Harrow Study Scope, in each pa se, three or four 
persons were the primary inforaants, i.e., the persons with 
whoa the interviewer spoke repeatedly and at length. 
(Depending on the circuastances, a few others were 
soaetiaes interviewed, albeit briefly.) T^e e valuator was 
always a priaary interviewee. The other priaary 
interviewees — typically, project directors, building 
principals or on-site program coordinator's — were selected 
for their degree of involvement in the evaluation effort or 
for* their significance .as prograa decisionaakers. in 
general, interviews did not extend upwards to 
adainis^rative or technical staff at the district office, 
or downwards; to teaching staffs The interviews, 
themselves, focused on evaluation work at' the program or 
building level, with a lajor topic being the evaluation 
events (meetings, conversations, etc.) in. which the 
.interviewees had participated* 

Because the interviewees were few and the interviews 
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were so often focusea on evaluation 'events,, limited data 
was collected on the evaluation 1 ^ wider content either 
within a school or within the host school district, it 
would hav# been helpful to learn sore about* the evaluator's 
tactics with other programs; customary evaluation practices- 
within a district; teachers 1 attitudes towards and use of 
evaluation; the pri nary concerns of school or program 
personnel during the years studied, regardless of whether 
these concerns directly related to evaluation; typical* 
decisionmaking procedures in the schools studied; district 
administrators* inducements (if any) to schools to use 
evaluation information; and district policies on evaluation 
and instruction* 

Soie of the preceding data could have come from 
existing interviewees* Other data could best have w been 
collected through interviews with persons above or below 
the thin .stratum of school personnel we contacted* fat the 
time the studies were conducted, it was not clear that this 
wider data would be useful* But, were we to repeat the 
studies, we would expand the interviewee pool to include 
informants at a broader range of levels within the schools. 

Me now would be inclined toward studies with a ^jj£oader 
set of informants and broader initial scope* One can gain 
a more thorough understanding of a school f s internal 
dynamics and external environment by contacting many * 
informants and by spending more observational time within 
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the school* A broader study, however, requires t broader 
clearances* \tl arranging for the case studies, every 
effort was made to keep a low research profile* In 
retrospect, it might jiave been better to try to negotiate a 
broader charter, for the research, even though -more * 
districts might have refused to participate* 

- 2ht Studies* Emphasis gnr Explainxncr TTSe * The case 
studies placed greatest emphasis on explaining the the 
conditions underlying instances of successful evaluation 
use* A major hope, after all, was to develop 
recommendations leading to greater evaluation in 
addition, it seemed intuitively plausible .that evaluation 
disuse was simply the "flipside" to evaluation use, that 
is, that evaluation disuse would be explained adeguately by 
the absence of those factors which led to use* 

In retrospect, it now seems more appropriate to 
conceive of instances of evaluation use as foreground 
elements set against a background of predominant disuse* 
The case study analyses satisfactorily explained how the 
instances, of use came to pass — through personal 
initiative jand consultative evalf^tion, primarily — but 
they provided less satisfactory explanations for the 
widespread disuse of evaluation* Why should personal 
initiative or exceptional evaluation activities be 
required? Why should their absence lead to disuse? The 
case studies did not move to these more fundamental 
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questions, in part because we aid not then recognize the 
need to seek explanations for both use and disuse* Today 
we believe that the t*x> phenomena need to be considered 

< 

separately* 

The Hole of 'Existing Theory . * The case studies 
relied minimally on existing theory. Instead, the study 
analysis sought to develop its own grounded theory (Glaser 
& Strauss, 1965) of evaluation utilization, deduced 
entirely froa the interview data* We we^e largely 
satisfied with this strategy in the case study phase of the 
research* 

^ Our later efforts have drawn more heaVily on extant 
theories of school organizational behavior (soae of which 
were only just 'emerging during the .years covered by the 
case study project) to help explain the lack of school 
attention to evaluation data* We feel today that a case 
study approach which attempts to draw upon relevant theory ' 
is quite desirable. At a ainimua, existing theory can> 
infora data collection, specify variables of interest, and 
suggest alternative interpretations of ^vents. * Analysis * 
should be grounded in and thoroughly substantiated by case 
data, and it should not bend the facts to fit 
presuppositions* But existing theory, properly 'used, is a 
wellspring of analytical concepts* 

Suam^rv Review * 1 

* The Evaluation Case Studies accomplished their 
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intended poirpc^s at a modest* total costw Even though 
relatively few informants were interviewed, the case 
studies yielded descriptions of local school evaluation 
'that were far more detailed than any others, then available. 

The studies indicated that persanal initiative and 
consultative, client-focused evaluation were associated 
with greater evaluation influence. This finding 

* 

corroborated contemporary research (Patton et al., 1975). 

The research methodology allowed school personnel to 
describe evaluation work in their own words, and it 
permitted an exaaiaatipn of school perceptions of 

IT 

evaluations influence, strengths, and 'deficiencies. 
Through open-ended interviewing, the studies uncovered 
quiet, incremental uses of evaluation information, lises 
that might not hive been detected by other means* 

The principal improvement that could have been made in 
the case studies was simply tp^ broaden the research — to 
interview more school personnel^ to spend more time in the 
schools, and to explore a wider range of issues. Had the 
scope of the studies been expanded, we might have learned 
more, more quickly, about the organizational and contextual 
influences upon evaluation. Neverthef|^»>wthe case 
studies, as conducted, were a substantial success; 
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Chapter 3 * 
'Evaluator ^ielg^Study 



RESEARCH DESCRIPTION 
The Evaluation Case Studies produced the analytical 
framework outlined in Appendix A, and isolated features of 



evaluation situations which' seemed especially significant . 
for evaluation utilization* The elements of the framework 
varied considerably in potential 'manipulabillty*, Some, 
such as the' evaluations mandatory requirements (element^ 
1*2 in Appendix A) or the relationship between the school 
and central district* administration* (5. 1) , were typically 
beyond an ^valuator^s control* But others/ especially 
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those related to the evaluator^s approach* could ^e^J^^^ 

purposely^ manipulated. „ The research report (A Ik in et al. # 

j - . ' *> • " . • • - . ! ' 

1979) r clearly- implied that evaluatcrs- should take into 

" ■ ' */ 

account factors such as those listed in the framework* 

*■ » *■ 

However/ it stopped short of formulating a recommended 
evaluation approach* 

Within the Evaluation Ose Project , however, poss&ble 
recommendations were being^ considered. One key to / * 
increased local eva luation' use appeared to be a 
consultative evaluation approach*, one' deliberately oriented 
to the peeds of specific local. program managers* t In 
addition* it* seemed thab successful evaluators had %adis a 
commitment to Care most about local users* -Although they 
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carried out" the officiaUy mandated evaluation tasks with 
integrity, their emphasises on the local- cop sequences of 
evaluation. 



The case studies suggested that a consultative, 
user-focused approach would stimulate local evaluation use. 
Yet there were iaportant guestibns left unresolved by the 
CSB case study data. For example', were there important 
organizational prerequisites to be met before a 
consultative evaluation approach could be effectively - 
implemented? The case studies had focused on the 
evaluators» work with programs, not on the organizational 

environment, so this question could not be answered 

* 

readily. Assuming that consultative evaluation could 
reasonably be recommended, was it applicable to all the 
programs with which the evaluator might work? Hight some 
programs rebuff the evaluator? 

The. seed for, ,% Fiel- tj Study of . Evaluators £t Work 

A detailed study of school program evaluators at work 
was needed to- examine the evaluation work environment and 
the nature of current evaluation efforts. - Even at a purely 
descriptive level, such a sjudy' would contribute to the. 
still limited database on evaluation practice. In 
addition, by working with the evaluators for an extended 

tod of time, the researcher might come to see the 
organizational panorama as the evaluators saw it. One 
could efj^lore with them the encouragements and constraints 
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that the organization placed upon their work, factors ^that 
might affect any atteapts at consultative evaluation. 
Horeover, the evaluators* interactions with program 
managers and staff could be directly observed. One could 
discover whether evaluators were already seeking to adapt 
their* work t v o local needs, if they were^one could see how 
their initiatives were received, f 

Field Study Hethods 

Overview 

The Evaluator -Field Study examined evaluation work as 
it unfolded and as it appeared from the perspective of the 
^ school program evaluator. The research relied primarily on 

on-the-spot' observations and informal discussions with the 

( 

evaluator and k colleagues. 

Three school program evaluators within the- same urban 

districtywere observed as they carried out various * 

evaluation activities. 4 ,The f ieldworker entered the st'udy 

site — a school district evaluation office — and spent an 

initial orientation period "tagging along" with the 

evaluator selected for the study, observing and discussing 

his or her work but also concentrating cn becoming familiar 

witji the work setting* '"'Thereafter, each evaluator 1 s work 

i 

was traced in more deliberate fashion, focusing on the 
stream of decisions and events surrounding various work / 
tasks. Critical events were observed as they occurred, 
•discussed with the evaluator, and followed to resolution. 
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Froti tha observations and discussions, an evaluator f s view 
of events was constructed* 

Selection, of Evalua tors ac,(| Stndv l Situations 

Criteria for selecting evaluator-stb jects were based 

priaarily upon a .conception* of the kind of evaluatocf or 

whoa the CSS Evaluation Use Project wished to, develop 

recommendations. 1 it seeaed ^ost appropriate to study 

in-house evaluators rather than external consultants* & 

consultative evaluation, approacl^aquiied extensiv^ 

interaction, between the evaluator and prdgraa personnel. 

In- house evaluators seeaed aore suited Jto this 

tiae-con spaing role.- " 

is a second criterion, subjects should be involved 
, , - « jf . 

vith prograa evaluations — not slaply district testing or 

• v ' - " %4*». 

the, like. Third, if aore than one evaluator was associated 

with a program,, the the preferred- subject was the senior 

evaluator, since he or she would be aost likely to make 

decisions about the evaluation's focus. Finally, subject 

' - ' M 

cooperation should be voluntary, not coerced. 

Resources and tiae dictated that no yore than two or 
three evaluators could be, studied and th^fc should be picked 
from no aore th^an twp districts. The choice between a , 
single district '.study 4 and"- a -two district study was 

* * * 

problematic. Concentrating all the effort in a single 
district would allow the greatest coverage of evaluation 
work in th« district. Splitting effort between two 



districts would allow cross-coaparisons of organizational 
e'nwironaents. Is it happened, chance events deterained the 
- ultiaate outcome: a single-district study. 

The research b egan * with a two-aistrict strategy* 
One was to be an urban district, the other a smaller 
suburban one* Effects o£*the difference in size, 
structure, and community environment were to be explored. 
The search for two districts proceeded along fairly 
inforaal lines, as is usually the case in sa^ll-scale 
gualitative studies (Bogdan s Taylor, 1975). Geographic 
proximity and the" existence of a likely contact person were, 
important considerations, Hetro unified School District, 
the urban district, was a clear choice based upon these 
criteria. The other site, santa Lucia (ADA approxiaately 
23-30,000), was a ^suburban bedrooa coaaunity. 

. Work began in both districts, but the santa Lucia 
coaponent of the study proved unsatisfactory. Contrary to 
initia-l understandings, *the subject in Santa Lucia normally 
did little program evaluation work. The research fifeld 
visits, fact, wete proapting* hia to consider initiating 
nfew evaluation** work that would not have occurred otherwise* 
With regret, the Santa Lucia involvement was terminated* 
It was impractical to locate a replafeement district at that 
point in the school year, and therefore all the reaaining 
work was, concentrated in the single urban district, Hetro 4 
Onified. 
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Metro On if led School District * 0 

Hetro Unified School Distriqt .was proiiised anonymity 
in this research effort, so exact details of Hetro 1 s -size , 
organizational structure, student or coinunitt population 
can not given* Hetro was a large ui^an district in 
California* It had taken soae recent steps toward 
decentralization, but decision taking on xaportant issues 
reputedly was still highly centralized. Its service 
population included students fro» all socio-econoaic strata 
and racial-ethnip backgrounds, Like several other 
California district s, Hetro was engaged in an evolving 
integration effort. In addition, its bilingual service 
programs were rapidly expanding in response to demographic 
changes and) state legislative requirements* Hetro 
aaintalfled in Evaluation & Testing Office, which handled 
evaluation related to compensatory education programs, 
coordinated district-wide achievement testing, and 
conducted special studies. " * 

Initial . Co ntacts in Hetro District 
In early Hay 1 * 1979, the CSE Evaluation^ Use Project 
Director, contacted the Hetro*s Evaluation 5 Testing 
director (an acquaintance) , described trhe purposes and 
■ethods of the study, and inquired about the district's 
willingness to cooperate. The letro Eraluation & Testing 
(E & T) Director was almost iaaediately redeptive to the 
field study idea. 
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The EST Direcyor arranged foe a nesting with Hrs, 
Elaine -Bow nan, an evaluation staff aenber fitting the 

evaluat.or-subject criteria. Few other staff in the EST . 

I 

Director's intfediate offices were assigned as program ' 
evaluators; most either 'held administrative positions or 
worked on special activities rather than program { 
evaluation, flrs. Bowman, however, was a recent addition to 
'the staff, hired expressly to serve as the evaluator of the 
newly created Preschool Language program for School Success 
(PLPSS). 

r The f ieldworker met with the EST Director and Hrs. 
Bo ; waan in their offices to discuss the proposed research* 
The EST Director remained supportive, and Hrs. Bowman 
seemed relatively unperturbed at the idea of being 
observed. The fieldworkfcr accompanied flrs. Bowman through 
the months of Hay and June, observing her work on the PLPSS 
evaluation. 

In the fall, after the summer hiatus, contact was 
reestablished^tiJ^ the Evaluation 8 Testing office. Hrs. 
Bowman had left Metro District (for a better position), and 
the fieldworker was directed to Hrs. Carrie Jenkins, an 
evaluation staff member within the Office's Compensatory 
Education Evaluation On it. Hrs. Jenkins -had an intriguing 
dual role as a "evaluation- adviser" for a group of Title I 
schools and as the sole evaluator assigned to a special 
district-funded child service program. It was .the latter 
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role that first attracted the Project's attention Mrs; 
Jenkins. However, as the research progressed, Hrs. 
Jenkins 1 , Title I duties took center stage in the research, 
and they were the focus of the final research report* 

Contact was made by phone with the Compensatory 
Education Eyaluation Onit f s supervisor, wh ©/"respond ed 
cooperatively — probably becaus^ he received assurances 
from higher Evaluation & Testing office adainistrators. 
''(Certainly, the previous spring's "Saoothly accomplished 
research activities greased the wheels for this secofta 
entree.) The supervisor, in turn, conveyed his approval to 
Srs # 'Jenkins,' and the f ieldworker called Jenkins to arrange 
a personal meeting. Hrs. Jenkins agreed to participate, 
perhaps with <soae trepidation although the research 
experience assuaged her concerns. 

Heanwhile, it was discovered tfrat the PLPSS was being 
evaluated once again by a new evaluator, Hs. Diane Gria es. 
Phone calls to the E & T Director and the Assistant 
Director (as. Grimes' supervisor) , secured their approval 
for a further study of the 1979-80 PLPSS evaluation. Hs. 
Griaes and the fieldworker aet in January 198D and 
inaediately began the new PLPSS research. 

Thus, ultimately, data was collected on three 
evaluators from Hetro Unified School District. 

Thf* Besearch Process 

The research details varied, guite naturally, from 
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evalnator to evalnator* Here the general pattern of the 
research is described* 

The, very first days in the field were devoted to 
getting acquainted and getting one's bearings* Typically, 
the first aeeting with the evalnator subject was something 
of a tw.o-sided interview: the evalnator assessed the 
fieldworker 1 s intentions, purpose, and style, and the 
fieldworker learned as aach as possible about the 
evalnator* s work and personal style. Of course, this 
mutual sizing-up was not completed at the end of the first 
meeting. Generally, the first few meetings had as part of 
their purpose the goal of acquainting the fieldworker with 
the evalnator »s work situation. 1 

For example, „in the PLPSS casH visits-to program 
classroons were quickly scheduled. And in the case of the 
Coap Ed e valuator, the fieldworker was soon scheduled to 
observe a routing Title I liaison visit and to take a* tour 
of the child service prograa with which the evalaator also 
worked. 

During these initial excursions, the evaluators served 
as guides to the workplace — suggesting interesting things 
to experience and making' introductions to colleagues. Much 
of the initial tiae was also spent establishing the 
research relationship, of course, the fieldworker was 
simultaneously becoaing aore faailiar with the evaluations 
job responsibilities^ and" current activities. 



With, the Comp Ed evaluator, Mrs* Jenkins, the 
fieldworker usually spent about a half day for each 
observational session* This corresponded to the way Hrs* 
Jenkins usually scheduled her visits to the schools* 
Interspersed with these half day sessions were occasional 
meetings at Jenkin f s office, to catch up on events that had 
not been directly observed, to plan future observations, or 
just to talk* < 

With Brs* Bowaan, the PLPSS*s first evaluator, the 
fieldworker was able to observe many of the evaluator 1 s 
activities during the six weeks in Hay and June 1979, when 
the bulk of the evaluation took place* Bowaan was too 
busy* then, fo£ lengthy interviews tut conversed at length 
with the fieldworker on the way to and froa aeetings and 
school visits* * * 

In contrast, Hs* Griaes 1 full year evaluation of -the 
PLPSS required a different research approach* Griaes put 
in long stretches of desk work in the 'office. This 
activity was difficult to observe unobtrusively. The 
fieldworker settled on a routine of frequent short 
interviews and phone calls to check -the progress of the 
evaluation* in addition, evaluation "events* were 
observed, for exaaple, the evaluator f s classroom* 
observations, testing activities, interviews with 
administrator?, and other trips to the schools* 

Table I sunaarizes the data on "contact hours* (spent 
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interviewing and observing) and number of field visits. 
Telephone conversations are not*' reflected in these figures. 

Table I 
ContaSts with Evaluators 



Evaluator 


Prograa 


Tiae 
Tr aae 


Contact 
Hours 


Field 
Visits 


Bovaan 


PLPSS 


5/79-6/79 


38 


11 


G rises 


PLPSS 


1/80-6/80 


46 


15 


JenJcins 


Coap Ed 


11/79-6/80 


70 


20 



Depending upon Jenkins' and Grimes' schedules, some 
weeks during the 1979-80 school year were very busy, 
involving several sessions with each evaluator; other weeks 
were quiet* The decision to dirdctly observe any given 
activity was a joint one, some activities the e valuators 
recommended as- likely to be interesting; others the 
fieldworker selected from the evaluator's upcoming 
schedule, based on *the contribution they night sake to the 
unfolding analysis of each case* The evaluators were given 
veto power over the f ieldwojrker* s suggestions, a power 
almost never exercised* 

It was sometimes possible to take notes during the 
actual observations — for example, during observations of 
meetings where other participants were making notes.. In 
any event, detailed field notes were prepared\ immediately 
following each observational session. * 4 v 

The rising Eval nations framework {Appendix A and 
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Alkin et al. , 1979, «Chap. ,9) identified situational 
characteristics to be observed. As the research unfolded, 
yet another list of potentially significant topics began to 
take shape. These were foraulated anj3 systematized in a 
series of draft topic lists te.g. , Appendix B). 

Working hypotheses about the important features of the 
evaluation work were developed as data were collected. 
These working hypotheses could then be napped against new 
data and refined, aodified, or replaced. 

the Besearch Re port 

■A 

Observations and discussions with the PLPSS and Coop Ed 
evaluators continued through the end of the school year. 
Following a coaplete review of the field notes, a draft of 
the research report was prepared in July and August. The 
three evaluators critiqued that draft in interviews with 
the fieldwcrker. Thereafter, a final report was prepared. 
It described the evaluation work carried out during the 
observational year, and it also discussed organizational 
characteristics which appeared to iapede evaluation 
influence. 
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r CRITIQUE 

Review o£ Field Study's Purposes 
The Braluator Field study was a •successor to the 
Project* s case studies, it was intended to supplement the 
case studies by collecting aore detailed inforaation on 
evaluation's organizational context and by allowing direct 
cbservation of evaluation activities. 

The case "studies had suggested that evaluation was 
aost successful when the eva,luator worked cOnstfltatively 
with local prograa administrators, attempting to address 
their interests and to involve them actively in evaluation 
planning and data interpretation. Personal initiative also 
was important to evaluation success, particularly 

♦ - 

evaluators* and administrators! joint initiative to create 

a consultative relationship, as veil as adainistrators 1 

resolve to use systematically collected information in 

decisionmaking. » 

But, was it realistic to advocate consultative 

evaluation and pergonal initiative as answers to the 

problem of evaluation under utilization? Would 

organizational realities allow consultative evaluation to 

be implemented on a broad scale? S could individual 

< 

evaluators or prograa administrators introduce this style 
of work on their own initiative, or wouia institutional 

changes be required? Answers to these questions required 

i , 

/ 



more data on evaluations organizational context and -on 
evaluators- interactions with school personnel. The field 
study's on-the-spot observations of evaluator work were 

« 

intended- to collect this data* 

There was a sabsidiary active for the Evaluator Fie^jJ 
Study. Quite siaply, it seeaed wise to look fibst-hand at 
school evaluation work. Although the caAsrtudies^had £one 
well, there were hazards to -basing analysis sc heavily on 
interview data. The. Evaluator Field Study, with its 
observational work, was to be a check on the case studies* 
findings. > t , 

Research Design Decisions 
Because the case studies had suggested steps 
individual evaluators Bight take to increase evaluation 

r 

usefulness, the field 'study sought to exaaine closely the 
wor£ of individual evaluators. This suggested a study 
fdcused on individual evalftators rather than a aoc:e' g|^eral 
study of an evaluation- unit f s activities. Saaple size 
would have to bp small* aost two or three evaluators could 
be studied in d'epth by .the soLe f ieldvorker. 

The evaluators could have been select e<T frca either 
the saae or -different school districts. There were 
arguoents for either strategy. As\s^a,ted drt the preceding *- 
research* description, chance events influenced the ultiaate 
research design, in which* three evaluators were studied, 
all froa the single district, aetro tJnified. 
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is in the case, studies) research access was negotiated 
, thro.ugh personal contacts, and subject participation' was * 
voluntary. Participant observation requires extensive 
•* subject cooperation over a long period of tine (in this 
instance, more than a. year) , a aking^ random site or subject 
selection quite |iff^cult. An effort was aad*e, however to 
avoid school district sooted for uniquely progressive 
evaluation practice. IWore typical setting was sought, 
^instead. N 

Standard participant observation aethods were 
•employe^, perhaps the only major aethdologicafl adjustment 
, * was that the office work being studied required 'more y 
r inte'rne wing than had 1 " originally beei*- planned (see Points 
of Concern , below). 

ffethodolaai cal Assessnent 
Descriptive Detail 
K Jt The field stud y're port described the subject 

^evaluators* work and characterized school atl&tudes towards 
evaluation. Two of the evaiuators studied worked full-tiae 

vith a single program (the* Preschool Language Program .for 

v 

School Success). Their work with this program was 
described chronologically, much as evaluation has been 
described, in the case studies. Program history and 
context were slfetched, and evaluation, fro* initial design 
£o final ( report in§? , was detailed. 

The' fieia study report did provide quite detailed data 



os the circuastancss, events, and deliberations tlvat 
affected the PLPS*«s evaluation design. Coapared with the 
evaluation case studies, aore inf or nation was presented to 
explain. why the evaluation haa £sken,the shape it had*' 
Some of this inforaation was collected through 
conversations ,witK the evaluator; soae, through direct 
observation of planning meetings and siailar events. 

PLPSS evaluation findings were described* First-hand 
observations and discussions allowed a comparison between 
what the e valuators said about a prograi, either to the 
fieldworker or to clients and colleagues, and what they 
wrote in their final evaluation reports. Fi&ally, some 
data was also collected on evaluations apparent impact on 
the prograa. 

The third s evaluator-sub ject was one of several 
"evaluation advisers" for the districts ESEA Title I 
programs. This evaluator provided technical assistance to 
Title I schools rather than directly evaluating the 
schools' educational effectiveness. The description of the 
evaluator 1 s work therefore was organized theaatically, 
around the several varieties a%technical service which 
were provided. ' 

Thro^hout the study report, vignettes from the 
fieldwork were used to illustrate situatidns encountered by 
the evaluators or to provide more concrete and vivid detail 
on critical evaluation events. The use -of vignettes^ was * 
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analogous to the use of direct quotations in reports of 
, interview studies* o 

In -the description of the PLPSS work, portions of the 
annual PiPSS evaluation reports were quoted* These reports 
ctnd other pertinent documentary materials were excerpted at 

length in appendices to the field study report. '{Segments 

° 

of these documents were excised so as not to disclose our 
host's identity.) 

The vignettes, narrative descriptions of critical 
events, and excerpts from documents together furnished 
evidential support for the field study f s descriptions of 
the Hetro school environment* This amount of descriptive 
detail was available, in large part/ because trhe field 
study examined contemporaneous events* Observations could - 
be made, partici^pants could be questioned as events 
transpired or soon thereafter, and miscellaneous memos and 
documents could be collected conveniently. 

The field study report contained few direct Quotations 
from participants, however* Conversations had not been 
tape recorded or transcribed by hand on the spot, so 
verbatim quotations were exceedingly difficult to capture • 
Paraphrases based upon the fieldnotes vere more readily 
constructed* 

* In general, the Evaluator Pield Study provided depth 

V 

of detail rather than breadtfi of coverage* That is, 
activities in which the evaluators participated were 



covered in great depth: observations, discussions, 

docuaent reviews, and seal- formal interviews with the 

evaluators let us scrutinize these activities closely. 

Other 'activities which, sight affect the evaluatGr, but in 

which the evalua'tor did not directly participate, were not 

as'^rell explored. Tor exaapB*, evaluation policy set by 

higher-level district management affected the evaluators. 

The fieia study explored the evaluators* perception of that 

policy, but policy- setting by «anagenent was not directly 

investigated. ,Siailarly, activities internal to a school 

could affect evaluation, but schools were not a^n 

independent focus of the research. The field study, 

y 

therefore, tsuly hewed to the eyaluator f s perspective on I 
events, with resultant advantages and disadvantages. 
Descriptive Accurac y 

Field study data came from observations of the_ 
evaluators at work, from a great many conversations with 
the evaluators, and from a audi smaller number of group 
discussions involving other school personnel, the 
evaluator, and the fieldworker. Proa this database, data 
could be substantiated to varying degrees. 

Events directly observed by the fieldworker were *ei\* 
docuiected, of course. And the fieldworker could compare 
his own understandings of these events with the 
interpretations provided by the evaluators or others.. 

Other data dealt with events or circumstances which 
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were not directly accessible to the fieldworker: comaents 
an history or current events which the fieldworker had not 
witnessed*; assertions that a practice was standard , 
custoaary, or aandatory; or discussions of policy, 
expectations,' or job requirements. This information could 
be exaained for internal consistency and plausibility, and, 
sin^e feuch topics arose frequently, there were aany 
opportunitie^ to ask for clarification or greater detail* 
But there vere limits to the confidence one could place in 
this data. 

^, 

Data quality depended greatly on the relationship 
"established between the f ieldworker and each evaluator. 
with the two evaluators whoa the fieldworker followed in 

, the 1979-8C school year, a solid working relationship 
gradually developed. By year*s end, these evaluators 
appeared coafortable with the research, candid, and 
forthcoaing. ie were coapletely satisfied with their 
cooperation. The (PLPSS) evaluator who was studied for 
seven weeks in late spring 1979 was cooperative on aast 
Betters, but less open than the- others. Iaportantly, froa 
the aany hears of observation and conversation, the, 

' fieldworker gained insights into the, evaluators 1 interests, 

v. • » - 

♦ ' predispositions and biases ~ thus, allowing aore informed 
assessment of th$ evaluators 1 coaaents. 

Without question, additional .data from a wider set of 
inforaants — -school-based personnel,* the evaluators 1 
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colleagues and' supervisors, ana higher * level district 
personnel — would have aided the study. There would have 
been aore opportunity to explore the contextual latters 
that evaluators described but that the fieldvorker could 
not verify independently. Because the fieldvorker did not . 
wish to dilute the study*s tight evaluator focus and his 
own close identification with the sub ject-re valuators, these 
other informants were not interviewed* 

Taking all these facts into account* our best judgment 
was that the study yielded atf accurate view of evaluation 
as it was known to the evaluators a^nd to an observer (the 
fielcfworker) * operating' fro* " t*he evaluate^* perspective*^ 
But it was also clear that the study offered a situated 
view of events* one limited by tire fie*ldwork f s emphasis on 
the evaluators f perspective* * 

Hature of the Field Stud; Analysis * 

There were district difference^ between the field 
study analysis and the evaluation case studied analyses* 
The case studies 1 analyses were constricted from 
interviewees 1 explanation? of evaluation influence in their 

programs — explanations- requiring a high level of ' 

% \ 

inference and synthesis on the intervie'we'es* part. The 

i * 

cross-case analysis then generalized froa these 
explanations. 

The field study analysis was constructed froa 
lower-inference datat observations, discussions of 
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individual events as they occurred, judgments of the 
typicality of an attitude or procedure, etc. This raw 
detail was assembled into a description of the evaluators* 
work and a summary analysis, of aspects of the school 
context that seemed to impede evaluation influence: loose 
administrative control of classroom instruction, resistance 
to outside intrusion into the school or classroom, and 
general hostility towards evaluation. 

The field study analysis compared the findings 
regarding Hetro with extant data and theory on school 
organizational behavior. Prom the literature, it appeared 
that^the organi£a£J.onal characteristics observed in Metro 
were common in public schools. According* to some 
theorists, these characteristics fit into a stable 
organizational pattern which was functional for school 
survival and growth* * 

The crux of the field study analysis, therefore, lay 
in matching the study da£a to existing theories of : school 
organizational behavior.^ Theory, in tarn, predicted 
obdurate resistance to evaluation activity and* evaluation 
infgraation, a prediction which conformed to the experience 
of Hetro evaluators in their efforts to increase evaluation 
influence. 

Generalizafcility , • 

The etaluator fieia study described the work of three 
evaluators in one district, Hetro unified. Two of the 
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three evaluators worked with the PLPSS, a unique preschool 
program within HetrO, and no claim was made _that others 
pursued similar work. 

The third subject w^s one. of several ESEA Title I 
evaluation advisers in aetro. Conversations with the 
evaluator and otehrs indicated that the evaluator studied 
engaged in a representative set of work tasks with the 
Title I programs. This impression was also supported by 
the User Interview surveys findings regarding evaluation 
services available to school administrative staff. 

The Evaluator Field Study also described school staff 

attitudes towards evaluation and evaluators. Observations 

\ * > 
of group interactions indicated that the characterizations 

were broadly accurate* Independent confirmation came from. 

the User Interview Survey, .which discovered school 

attitudes consistent with those repor+ed in the Evaluator 

Field Study. 

., The Field study did not clai» inherent- 

generalizability beyond Hetrq Unified. But the study's 

findings werfe consistent with patterns of organizational 

behavior observed in other school systems, as described in 

the ^organizational theory literature^ Therefore, there was 

reason to telieve that the Hetro findings might be 

appl icab le^lsewher e. 

Points of Co nee rn 

Lack of Interi m Reporting . Comprehensive analysis 
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was deferred until the sumaer of 1980 , when a first draft 
of the study report was prepared* Before then, analytical 
theaes had been discussed informally within the Project, 
'but no written synthesis had been undertaken. In 
hindsight, it might have been helpful to have written an 
interia research report, perhaps in Harch or April. If 
such a report had been written, subsequent data collection 
might have been refbcused in drder to explore aolTe 
coapletely the eaergent analytical theaes. 

Fieldworker Fatigue. The mechanics of participant 
observation were aore tiaeconsuaing and fatiguing than had 
been anticipated. Fieldwork sessions were exhausting 
because of the concentration required to collect data and 
simultaneously aaintain a neutral, unobtrusive role. Field 
no*e preparation was a lengthy process, even when the notes 
were dictated' rather than directly typewritten* Hote 
preparation easily required twice as auch* time as the 
observation sessions* 

BecaUse data collection is so tiae consuming, it 
becomes very tempting to cut corners, analysis is 
deferred. One procrastinates about coding field jaotes. 
Hore insidious, thaugh, is the way that one may 
unconsciously siaplify and routiniza data collection by 
repeatedly observing the same types of events, talking with 
the saae people, and visiting the same schools. 

One needs the opportunity to step back ffonrthe 

s 
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harried life of the aata collector in. order to think 

> 

carefully and creatively about .the research* In future 
participant observation studies, we would try to structure 
more opportunties for reflection and planning, by^ 
scheduling 'planning and analysis weeks, interim reports, 
and periodic rap sessions with, colleagues, for exaaple. 

Ihg, Problem of the. Office Environment . Huch of a 
progran evaluator*s work takes place in an office setting, 
and there is a good deal of quiet desk-work or telephone 
conferencing. It is extreaely difficult to observe these- 
office activities unobtrusively, is the fieldworker to sit 
nearby? If , so, where? What is the fieldworker to do while 
others in the office are working? -How can the fieldworker 
collect data about telephone conversations without becoming 
a nuisance or an embarrassment? There. are no easy answers 
to these questions, when the evalbators held meetings or 
made .visits to schools, observations could be made more 
comfortably. Then, the fieldworker was accepted with » 
equanimity But in a general office setting, the 
f ieldworker* s constant presence was troublesome. 

The compromise solution was to observe scheduled 
meetings and school visits, but to visit the evaluation 
office setting primarily for brief drop-in "calls before or 
after scheduled observations or for scheduled discussions 
with the evaluatbrs. Two of the subjects had heavy 
schedules of aeetings and school visits, so considerable 
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observation; work was done# One e valuator had more office 
time; interviewing Was more frequent in her case* A major 
difficulty with this approach was that the evalator f s 
impromptu meetings in the evaluation office were not 
scrutinized* 

Alternatives to traditional participant observation 
methods might be more suited to a professional office 
environment* For example, subjects might complete daily 
activity logs or they might be asked to dictate a daily - 
journal, with their dictated tapes being collected and 
transcribed weekly. Observations and interviews could be 
interspersed with these other data collection 'methods, as 
appropriate* It seems imp&rtant, in any .event, to 
experiment with new methods for studying office work. 
The , Effect iveness of Person-Centered Ethnography 

The Evaluator Field Study . was person-centered in that 
it examined the worklife of selected individuals. 
Ethnography and participant observation more often explore 
activity within a physical setting (e.g., a hospital or 
prisoh) or social group (e.g., a youth gang) , although 
precedent for person-centered ethnography exists. (e#g#, 
Wolcott, 1973.) 

In retrpspect, we have some doub*s about the 
efficiency and efficacy of person-centered ethnography. 
The method is forbiddingly intense, observer and subject 
both value their privacy and find constant partnership 
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difficult* Tacit understandings limiting observational 
frequency emerge, and the observational work is likely to - 
be less complete than the researchers originally nay have 
envisioned* 

To the method's credit, strong working relationships 
can emerge between the ethnographer and the subjects, 
which nay increase subject candor* To a degree, the 
observer and subject Bay become co-investigators. But 
there are disadvantages, tpo* The f ieldworker f s close 

4 

association with the-subjects is" very apparent, and the 
fieldworker aay find it difficult to becoae a confidant to 
other persons in the setting. Also, it aay be difficult to 
divert research attention to persons or settings detached 
froa the original subjects, should the need arise. 

In the Evaluator Field Stiidy, for exaaple, once the 
research ccnaitaent was aade to the e valuators it would 
have been difficult to switch tactfully froa 
evalua tor-centered observation. Yet, the incoaing data 
aight have justified greater attention to school-based 
evaluation consuaers,' to the evaluators* colleagues, and to 
top administrators in the district office, in the field 
study, therefore, we encountered a "problea similar to that 
encountered in the case studies, naaely a sense that the 
inforaant pool could have been expanded usefully. In the 
field study, however, the initial research specifications 
(viz. person-centered -ethnography) iade it guite difficult 
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to include additional informants, if the research had been 
focused on the topic of evaluation linkages with schoo^. 
decisionmaking, rather than on the evaluators per se, 
then an expansion of the study to include these additional 
informants might have been accomplished more easily. 

Summary Keview 

As hoped, the Evaluator Field Study yielded detailed 
information about the work context of local school 
evaluation, and the study was a provocative test of the 
earlier case studies 1 explanations for evaluation 
utilization. * Confronted by the Setro data, we were forced 
to temper our faith in consultative evaluation' as a remedy 
for evaluation underutilizatio n>~*he field study , s 
greatest asset. was that it offered this opportunity to 
witness evaluation in the makinq. 

Our principal regret was that data collection was 
confined, to a degree, by the evaluator-focused approach 
that was taken. Greater flexibility to follow the emergent 
research leads would have'been helpful. In any future 
applications, we wo,u!d be Inclined to experiaent with a 
broader range' of data collection activities, although we 
would not wish to forego the observational work, which was 
so useful. 
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* Chapter 4 

User Interview Survey' 



BESEARCH DESCRIPTION 
The User Interview surrey, described here, examined 
the views of local school evaluation users: school * 
principals (and aid- level staff within schools. The 
interviews sought to discover the primary issues, 
decisions, or concerns occupying school staff; to determine 
e if evaluation information was relevant to, -or influential 
in, school decision making related to t&ese primary 
concerns;- to acquire inforaation on the uses to which 
evaluation information was put; and\to obtain staff 
opinion^ afaorat current evaluation services. 



One motivation for the User IntervJ«c Survey 
concern that the previous researchSrf^ght have ^erstatefc 
evaluation's importance in the schools, ^?he initial CSE 
case studies had ficused tightly on evaluation activities; 
froa the studies it^vas difficult to place evaluation in 
proper perspective. Was st^ff a^fetftion mostly directed 
elsewhere and only occasionally turned to \ 
evaluation-related matters? when" program decisions were 
made, how much discussion and deliberation took place *and 
what role did evaluation data play? Interviews could 

r 

address these questions. * 

In addition, the interviews ccuild reach a larger, more 
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representative, .sample of school .personnel that had the 
case studies^ Staff views on evaluation strengths and' • 
weaknesses, on aval nation. influence, and on' potential 
evaluation iap^oyea§nts- could be systeaatically explored, 
ind by-conducting the survey in the. satfe district chosen 

for th& Evaluator Field-Study, the evaluators 1 and the' 

{ ' ' • 

local schaol users 1 perspectives could be compared^ 

* S tudy Procedures 

Selrittinq the Fesponde^b Sample 

Defining the Pop ulation , . selecting a respondent 

f c 

sample required ope rationalizing the tera "local school 
evaluation users". Several saapling decisions were quickly 
. deteraihed by the research cir coast an ces. Because the User 
. Interviews were intended to coipleaent the EValuator Field 
Study, Metro Oriified School District staff wooia be * 
saopled. However, to cover Hetro, an urban district, giiite 
a nuaber of interviews r would be'reguired. Hith the 

* * 

resources available ,-fletro alone coyld be studied* 

The project 1 ^ prior focus on evaluation in -specially 
funded programs directed attention to users connected nith 
such programs. Hore 'than this, r hov.ever, there was the 
simple fact that Hetro did • little . program evaluation * 
except, of specially funded prograas. Thd study, was 
further United to schools receiving Title T funding* One 
could be 'sure such schools had experienced evaluations , 

■ » v- e 

since Tii:le I # requires then, .and the Title I program o-ffers 



a flr#£ gopl of schools from which to sample* 

« r mi 

( - Three individuals were interviewed at each school", 
site, in part to obtain overlapping responses from multiple 
.informants in order to "triangulate" the data' but also" in' 
past because ^fletro* s Title I schools seemed to, have 



multiple important decision makers .-(Baillak, 1980). The 
school principal was interviewed in every case. In 
addition, two other persons holding influential positions 
relating to the school 1 s programs were selected. Various 
y school personnel fit this description; a working list of 
all acceptable job titles was developed. 

As bne of the two additional interviewees, the ,staff 

Zber^ coordinating the school's special programs was 4 
ected. (This was usually the Title 1 Coordinator, 
Special Program Coordinator, or an Assistant Principal.) 
The final respondent at each school was to be sooebne 
involved "in administering the special program, although not 
necessarily at a seijior level* In a large school, this * 
individual's duties ~might be entirefly administrative. Jn a 
small school, it* was often necessary to interview/a 
Resource Teacher, Curriculum Supervisor, or Bilingual 
Coordinator — individuals who usually had teaching 
. responsibiH.it ie's in- addition to their administrative 
duties. " ♦ ^ 

Contacting the jSch ool District . ^The Metro 
superintendent was contacted. He approved the project, 
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comnitted the District to 'participation, and directed the 
Evaluation and Testing Office • {E 5* T) to assist in saaple 
selection. 

An oversize saaple was initially selected , to allow 
• ? * 

for attrition. Twenty-eight Title I elementary schools * 

were randomly selected, although a final saaple of only 

twenty ^schools was desired. The district compiled, the 

desired saaple/ which, included schools from all geographic 

areas of the district as well as. schools of diverse size 

and ethnic composition. Each school principal received a 

letter from the Superintendent informing, hia that his 

school had been selected. The letter briefly described the 

study, endorsed "i'ts purposes, and vouched for the 

researcher *s credentials — but also aade It clear that 

school participation was completely voluntary. 

The school principals were telephoned, feainded of th<^ 

Superintendents "letter, and asked when a research teaa 

member aight conveniently conduct the interyi^ws. Initial 

calls were aade in the sequence in which the school names 

appeared on the list received from the District. Heturn 

- < * » 

calls were often necessary, however, in order to reach the 

principals*' all but two of the principals contacted were 

willing to participate, and saaple selection was halted 

once 20 principals had commit-^d, themselves and their 

schools to the study. This saaple was was augmented with 

two. additional schools, selected froa aaong those served by 

i 
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the Compensatory Education evaluation adviser studied in 
the Svaluator Field Study. Thus, ultimately, 22 schools 
participated in the study. 

(7 , • 

* Interview Strategy &nd Format 

Strategy . To- place evaluation in context, one must 
recognize that school-site decision makers have many 
pressing concerns , other ^tl^an evaluation. Humerous issues 
compete for staff attention, and it seems likely that aany 
of tmese escape exaaination in the typical school* prograa 
evaluation. Therefore, to t4ke the local informants 1 point 
of view-, the respondents were first* asked to describe 
recent significant prograa occurrences. Hext, the 
prograa 1 s* recent evaluations were discussed. Finally, the 
interviewee was to expatiate on evaluation's general 
usefulness, strengths, and problems. 

The rationale for this strategy wis siapie. If the 
interviews had opened w^th dintect' questions about 
evaluation, they might have led the informants to overstate 
the- iaportance of evaluation-related issues. Therefore, 
the interviewees were to identify "signif icant program ' * 
occurrences" first and discuss evaluation later. Also, 
hinging much of the. interview on specifics — significant 
program occurrences and recent evaluations — helped avoid 
th£ generalities that plague abstract discussions of 
evaluation's virtue^s. Faults, and Impacts. 

Shy the phrasing "significant occurrences" rather 



than, for example, "signif icant decisions" or "significant 

concerns"? "Decisions" seeaed tao narrow, and perhaps even 

unrealistic. Following Weiss (1980), it appeared that' 

local school personnel aight h^ve difficulty thinking of 

theaselves ^as decision Bakers and identifying decisions 

they ha'd made* feiss argues that, in bureaucratic 

organizations, policy actions often are not decided but 

rather accrete in a gradual flow of "snail uncoordinated 

steps taken in aany offices — by staffs who have little 

awareness of the policy direction tba't is being promoted or 

the alternatives that are being foreclosed" (p. 382). 

If decisions seeaed too narrow and idealized, then 

concerns seeaed too negative and issues too auch in the 

reala of ideas or attitudes, possibly not yet ractualized. 

The tera, occurrences, was. a coaprotaise. A "significant 

occurrence in tbe life of the prograa" seeaed feore likely 

to be soaething that inforaants could identify, discuss, 

and analyze, it connoted a change or departure froa the 

ordinary streaa of activity in the school — an opportunity 

* — 
for influence, soaething that evaluation might {or aight 

no*) have affected. 

\^ By having respondents discuss these self -selected 
occurrences, one could investigate several aatters: what 

A 

the respondents felt were important decisions or events; 
how the school went about handling the occurrences; and how 
evaluation or other inforaation sources were used tb deal 
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4 with the occurrences* 
Format 

Pat a Collection in Perspective . The Questionnaire- 
Jawboning Continuum is one construct for thinking about the 
use of structure in data collection*** At the questionnaire 
end of the continuum, data collection is quite structured* 
All respondents face the same questions , which appear in 
predetermined order* In a selected-response questionnaire, 
idiosyncratic responses are not permitted. The respondent 
cannot volunteer information beyond that asked for in the 
instrument, and the data collector cannot tailor the 
interaction to the individual respondent, while this data 
collection method offers comparability across subjects, its 
sensitivity is limited to the choices within the 
queJSlnnaire. 

"Jawboning", defines the other extreme of the research 

continuum, in jawboning, neither questioner nor respondent 

is bound by external structure. In fact, the notion that 

• * 

one person is the guestioner and. the other the respondent, 
is misleading* Jawboning is more nearly an unstructured 
conversation between two persons. Heither party has a 
specif ic. agenda, and the conversation Is guided only by its 
own internal logic. Jawboning is rich in detail, since the 
participants are free to exhaustively explore .any given 
topic. However, since each . conversation takes its own 
v unique course, data from "'jawboning" lacks comparability 
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from subject to subject* 

Between the two extremes, there are other data 
collection options* For example, in a "standardized 
interview*, questions can be carefully scripted, but the 
interviewee can be allowed open-ended responses* or, as 
another option, an interviewer might be allowed to conduct 
a seemingly freeflowing conversation with the subject, 
after which the interviewer might complete a structured, 
forced-choice questionnaire reporting on the discussion* 

The Topic-centerqd Interview * Initially, a 
standardized interview providing for open-ended responses 
was considered. This choice ultimately was rejected as too 
rigid to fit the diverse "signif icant occurrences" which 
the respondents might choose to discuss* Instead, a 
topic-centered interview format was chosen, such a format 
modestly structures the interview — by outlining the 
topics to be covered — but it leaves question phrasing up 
to the interviewer. The respondent is almost entirely 
unfettered , except as the interviewer m,ay refocus the 
respondents remarks or move the discussion along to other 
topics* Thus, the topic-centered interview offers great 
flexibility within a guiding framework. 

Pat ton (1980) terms the same method the "interview 

guide" approach, 'and he states its function rather well: 

An interview guide is a list of questions 
or issues that are to be explored in the 
course of an interview, in interview 
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guide is prepared in order to make sure 
that basically the same Information is 
obtained from a number of people by covering 
the same material. The interview guide 
provides topics or subject areas within 
which the interviewer is free to explore , 
probe, and ask questions that will elucidate 
and illuminate- that particular subject* 
Thus, the interviewer remains free to build 
a conversation within a particular subject 
area, to word questions spontaneously, and 
to establish a conversational style — but 
with the foc.us on a particular subject that 
has been predetermined* (p. 200) 

'The Interview Survey*s topic guide is displayed in 
Appendix C. The brevity, indeed .the almost skeletal 
quality of the guide, underlines the key ramification of 
the method: interviewer training must be comprehensive an 
thorough* The training, with its supporting materials 
(appendices D 5 B) , inculcates in the interviewers the 
purpose of the interviews, specifies the information which 
should be sought out under each topic, and prepares the 
interviewers to secure useful data. The brief written 
guide simply cues the interviewers, reminding them of the 
interview structure* 
Interviewer Selection and Training 

The Research Team* The interviewees were drawn from 

a group of advanced students enrolled in a UCLA graduate 

seminar on evaluation utilization. All students received* 

five weeks of training* Five interviewers and five * 

"validators" were selected to be members of the research 

team, together with the Project director. Those students 

4* 



\ 



who had deaonstrated the greatest interviewing proficiency 
during training, and who had prior public school 
experience , were designated as interviewers* The 
validator's role is discussed in a subsequent section. 

The Training P roqraa . Training involved four 
phases: (a) familiarizing- students with evaluation 
utilization research; (b) developing general interview 
skills; (c) familiarizing students ,«ith Hetro District' 
adainistration, organizational structure, and terminology; 
and (d) training students in the specific purposes and 
procedures of the User Interview Survey. 

All the students r^ad and discussed Using 
Evaluations : Doe? Evaluation Hake a Difference ? (Alkin 
et al., 1979). All had previously read Otilization- 
focus^d Evaluation (patton,* 1 978) . In addition, tfcey 
reviewed other articles on evaluation utilization, 
including works by Weiss, Caplan, Braskaap, David, and 
others. 

Interview skills training was conducted by a 0CLA 
faculty specialist; in qualitative aethods. Training i 
sessions involved lectures, videotapes of aodel interviews, 
practice interviews, and discussions. 

To faailiarize the trainees with the Metro research* 
context,/Daillak, the investigator in the? Evaluator Field 



Study, lectured on the organization of the Metro Evaluation 
and Testing Office, and onthe evaluation activities 
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commonly found in the schools. A glossary of common school 
terms relating to special prograas and evaluation was 
presented* 

Finally, the training focused on the detailed study 
procedures. Supporting materials were developed, including 
an interview topic description, aock interview narrative, 
a one-page topic guide, and suamary data reporting foras 
for interviewers and validators. 

The interview topic description explicated the topics 
to be covered in the interviews (Appendix D). The aock 
interview narrative was a, facsimile transcript of the 
interviewer portion of an interview (Appendix E) • The 
topic guide was a one-page topic summary, to which 
interviewers were to refer during the actual interviews. 
(Appendix C) . The summary data reporting forms (Appendix 
?} will be described in a subsequent section. 

The trainees, together with outside experts, helped 
revise the training materials, in addition, interview 
procedures were pilot tested in two schools, and training 
materials were modified, as necessary. For example, . 
improvements were aa^e in topic- sequencing, suggested 
phrasing of interview questions, etc* 

The trainees reviewed and discussed transcripts and 
audio tapes from the pilot interviews. Questioning 
strategies and question phrasing were considered during 
these sessions. The relative importance of .each topic was 
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discussed; interview time allocations to the topics wer* 
suggested, along with procedures for keeping the interviews 
on target* 

Before his or her first schocl interview, each 
interviewer conducted an hour-long simulated interview with 
a study teaii member who role-played a school decision 
maker, mimicking the kinds of responses that had been 
encountered in the pilot interviews* During the interview, 
the nock interviewee took notes on the interviewer *s 
questions, techniques, successful and unsuccessful tactics, 
and on information which the interviewer had neglected to 
request* At interview's end, interviewer and^ mock 
interviewee discussed the interaction, and the interviewee 
suggested areas for improvement* 

The tapes from the first actual interview were used to 
train interviewers in data summer izat ion. Bach interviewer 
listened to tye tapes from this interview and completed the- 
data summary reporting form. The interviewers then met 
jointly to discuss the summaries, ind, based upon the 
exercise, improvements were made tp the reporting forms. 
Just as importantly, however, group discussion helped to 
standardize the data summarization process. 
The Field Interview s 

Sch^d . ulinq interviews . The interviews were 
scheduled by telephone two or three weeks before the 
interviewing was to begin. Each school was telephoned, and 
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the proposed interview procedures were discussed with the 
school's principal. Principals were told that three 
one-hour interviews were desired at each school. They were 
asked to identify two other staff aeabers who were school 
level evaluation users, as defined earlier, and a tentative 
date and tiae for t_he interviews were set. 

The three interviews at a school were scheduled 
consecutively, with a 15 ainutes break between each. The 
•schools were asked if they could arrange a quiet location 
for the interviews, .school personnel were very 
cooperative. 

appointments were reconfiraed by telephone one or two 
days in advance. If the naaes and titles of the two' 
subordinate staff interviewees had not been obtained 
previously, they were, obtained during this second telephone 
^contact. At one or two schools, a scheduled interviewee^ 
was unavailable the day the interviewer arrived, and the 
principal had selected an appropriate substitute satisfying 
the respondent selection criteria, in one of the 22 
schools, chance circumstances on the day of the interview 
resulted in only two, rather fhan three interviews being . 
conducted. 

Conducting Interviews. Interviewing proceeded 
saoothly. interviews usually . took place in private 
offices. The first interview at a school was always 
conducted with the school principal. Before' that interview 
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began, the rest of the day f s schedule was always 
reconfirmed. 

At the beginning otf each conversation, the 
interviewers indicated that they wished to tape record the 
interview to ensure accuracy hnd to facilitate future 
analysis. There were no objections to this, although a few 
respondents asked that the machines 'be turned off 
momentarily during the one-hour sessions. 

Ds\ta analysis 
Developing the mit ia l Data Base 

It was theoretically possible to transcribe the 
interview tapes and use the written transcriptions as the 
base data* However, the 65 hours of tape would have 
produced an unwieldy voluae of transcripts and required 
substantial clerical support. Instead of , being 
transcribed, the interviews were summarized on standardized 
reporting forms (see Appendix F), The -suaaary forms 1 
structure paralleled the interview topic guide and 
interview topic description, . 

As soon as possible after finishing the three 
interviews at a school, the interviewer completed the 
summary forms, one for each interviewee. The interviewers 
referred to their written notes and listened to the 
interyiew tape as they completed the summary. Then they 
listened again in order to hand transcribe quotations that- 
seeied particularly valuable. The final sunfflary. forn, 

f 
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therefore, contained five or six pages of narrative*, •* ' 

organized by topic area, and as aany as five or ^six 

additional pages of selected quotations* 

. Validating the Initial* Data Base 

. - — 

Several strategies were employed to increase the 

interview suaaaries* accuracy* As described prei£ousiy, 

tape recorders were used in each and every interview. 

There are arguaents both for and against interview taping. 

) 

In its favor, taping: (a) lets the interviewer concentrate 
on questioning rather f|han on transcribing the respondent's 
answers; (b) lets the inlWviewer focus his or her 
detention on the respondent and aaintain a aore natural, 
conversational interaction; and (c) peraanently records the 
study 1 s raw data. - 

Using the taped interview record, it was possible to 

if 

obtain a second, independent summary of each interview and 
thus help confira -the interviewers 1 suaaaries. After the 
interviewers 1 suaaaries had been coapleted, J&h& interview 
cassette tapes were given to a "validator. n Working from 
the tapes alone, the validator coapleted suaaaries exactly 
like those used by the interviewer. Validators listened 
fully to each tape before beginning their suaaaries; then 
they replayed the tape tfhile* coapleting the suaaary foras. 
Validators also identified and transcribed key quotations, 
listening to the tape once aore, if necessary. 

Interviewer and validator suunaries were compared, and 
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a procedure was developed for resolving^ inconsistencies A 
panel, consisting of tire inter viewer^ the validator, *and a 
.third' research team member, examined both written versions 
of the' interview and, if necessary, listened, to the 
interviews? tape to resolve disputed issues* 

ft 

Few^dif ferences were found beyond variations in th£ 

* *v • 

degree of detail with which observations were reported * on 
the summary forms. There were two' or 4 tbree instance? in 

^all in which, an interviewer and validator'reported 
contradictory information. He listening to the tape 
Recordings easily resolved the differences. The small 

'number of discrepancies *and their simple resolution were 
encouraging indicators o£ summary accuracy. 

' Nevertheless, a second quality check was. introduced 
(on a sampling basis).' The interviewees at the second 
school visited by. each of the five interviewers were asked 
to comment of the summaries of their interviews. Copies of 
thp interviewer summary -forms. were mailed to the 
respondents, and folldwup phone calls' were made a week 
after the mailing, reminding respondents to return the 
summaries with their comments. Ten out of 15 summaries 
we're returned* (ft« seems plausib le ,t*at interviewees who 
disagree^ wit ft the sifmm&ry would be more likely to return 
coirme&ts^) Tour of^tlie". '^pvT res pondejits" made no correct ions; 
Twenty-six comments were raad£ on tho^ Remaining sir" forms. 
The respondents 1 had f*w substantive differences with 
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th^> summaries. In most cases, the respondents simply took 
opportunist© provide additional (i#e», supplemental 
5orative) information not mentioned in the original 
fonvecsations. 
Data Synthesi s 

Data -synthesis entailed group discussions, simple 
quantitative tallies, and a procedure for "interrogating* 
^ the data base, 

9 

Group Discassi ops and Quantita ti ve Tallies . The 
( initial data synthesis began with wide-ranging discussions 
within the research team (i.e., among the principal 
investigator, five interviewers, and five validators). The 

i 

team met weekly after completing the interviews and " 

validation* Team discussions touched upon many topics, 

including what constituted "significant occurrences" for 

the respondents, what data sources seemed most important to 

the interviewees, and what Interviewee reactions were to 

evaluation data* % 

After three group meetings, each team member drafted a 

report based solely on the interviews he or she had 

conductpd or validate-d. Several tentative analytical 

themes emerged from the reports. These themes then were 

♦ «* » 
critiqued in a t conference telephone call with outside 

colleague (Siohael Patton). 
■% 

Some simple quantitative analyses were carried out 
immediately, For'example, the respondents 1 job titles were 
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categorized and tallied, other tallies provided a quick ' 
descriptive look at the kinds of "occurrences" discussed by 
the interviewees. 

Psvng Case Experts. As the research team 
discussions progressed, questions became more complex, 
requiring more time -consuming study of each written 
iflterviefl'' summary than had been necessaTy for the sample 
tallies, to expedite this, each team member assumed 
responsibility for the interviews from approximately four 
schools. In effect, each team member became the case 
expert forr*about a dozen interviews. As far as possible, 
-team Hembers were assigned intervie ws<. witjx which they were 
already familiar, namely, intervie w$ t he y~h a<i conducted or 
validated. The case experts became sufficiency familiar . 
with their assigned interviews that they could quickly. ' 

• ^ * 

recite or locate relevant facts in the data s.'ua&aries.* 

. •* " ". 

IX this point, the data analysis discussions began to 

* 

focus more' tightly" on a 'lifted number of themes. Each 
team member *took charge of one or more of the tt hemes, by 
questioning tallow team, members on relevant points during 
group meetings or by preparing short written questionnaires 
to be completed outside of the meetings. Acting as case 
experts, the other team members marshalled facts and 
quotations and provided £age citations to the written 
summaries. The "theme ^leaders' 1 drafted analyses of their 
chosen themes, ^*hich were 'then reviewed by tl££ full 
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research teas. The most promising - drafts were expanded 
into Project working papers' ' ~'\ 
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CRITIQUE 

Review of thte Interview Serve y Purposes 
The Interview Survey was designed to provide a" broader 
perspective on school staff attitudes towards evaluation > 
zyt thereby coapleaent the ^valuator Pield Study's work in 
Hetro Onified School District. The Evaluator Pield Study 
was expected to provide only partial knowledge of 
school-level attitudes towards, and uses of, evaluation 
inf ormation, because the field' study was so tightly focused 
on evaluatcrs rather than school Site personnel* To 
explore school perceptions aore fully, .we wished to 
interview at some of the schools served by the e valuators 
under studyiA And to place these interviews in proper 
perspective, we hoped also to interview , personnel in other, 
similar 'schools* * 

An interview survey of school* level staff seemed an 
ideal sti^tegy* . with proper design,* the survey results 
would be representative of Hetro staff Appropriate 

* < 

questions could ba fornulated based upon the. case study 
findings and the. Hetro fieldwbrk. And the data froa 

'4 - * » 

schools served by the field- study's evaluator-^ub jects 



» 



could be exaained /Totfits compatibility with the fieldwork 
findings. ) • 

Research Design Decisions 
Although the Interview Survey was* to complement t'he 



•84 

87 



*7ield study it was also to be defensible on its own as a 
research investigation* Therefpre, every effort was made 
to survey a representative sample of ESEA Titlje I schools 
in Hetro and to collect comparable data from ^11 the 
mtervie wees. 

As anticipated, the Evaluator Field Study influenced 
Interview Survey Design. Proa the fie Id work, it appeared 
that instruction was seldom an issue for school-wide 
discussion, planning, and decision making. Administrative 
personnel in the schools seemed detached from instructional 
decision making and, therefore, not always interested in 
evaluation's findings about instructional eff ectiveness. 
In order to explore these indications, school planning and 
decision making was added as topics for the interviews* 

The, rationale for the interview guide has been 
described earlier in this chapter and will not be repeated 
here. *The choice o f £nterv iew length (one hoar) was a 
compromise: long enough to establish some rapport and to 

move beyond pat. answers to questions; short enough not to 

« 

inconvenience the interviewees significantly* 

Several* considerations prompted the decision to tap^ 
record the ~intervie ws. in a one-hour interview it seemed 
unlikely that taping would make the subjects much more 
reserved or cautious than they otherwise would be — thus 
vitiating the major argument against recording* In 
taping* s favor was the promise of having a permanent 



verbatim record' of .the interviews. Full transcriptions 
were too costly , however* . fts a compromise strategy for 
extracting quotable material and a written interview 
record, it was decided that narrative summaries would be 
prepared by the interviewer and illustrated with quotation? 
selected from the tapes* * ' . 

Het hodoloa^ ca ^ Assessment 
Descriptive Detail 

The interview Survey's descriptive detail must be 
examined from two perspectives. First, one must consider 
the raw detail developed in the interviews themselves. 
Second, there was the detail transmitted in the study's 
analyses.'"' * ~ 

Baw data compr rehensiveness and detail varied widely 
from interview to interview and from topic to topic within 
each interview. Interviewer training .had included a model 
allocation of time to topics, but a review of the tapes 
showed that the time allocations varied significantly. In 
some cases, „the interviewees appeared to pull the 
conversation to a certain set of topics; -other timers'", the 
interviewers appeared to guide the interview 
idiosyhctatically. Irrespective of time allocation, detail 
varied depending upon the ^interviewer «s skill and the 
interviewee's talent as an informant. Overall, the 
interviews covered the^ desired topics, but not always in 
the desired detail* or proportion. 



The study analyses drew upon the raw interview data in 
different ways. For example, aini-case studies-were 
prepared for two schools (the schools in the sample that 
were served by the Compensatory Education education adviser 
from the Evaluator Study), These mini-case studies offered 
the aost descriptively detailed presentations of interview 
data* They ^re developed froa an intensive examination of 
the six hours of taped interviews in these schools. 

Host of the regaining analyses relied upon the Written 
interview suaaaries and the « case experts 1 recollections of 
the subset of tapes each had auditionedf Several working 
papers were prepared. Two were classifications of $ 
Tntervie wees, progr aas,^an(T~the n signif icant occurrences"" 
discussed, by the interviewees. These two papers were 
prepared froa the' interview summaries and did not aake use 
of direct Interview quotations, i third paper summarized 
Hetro f s aajor evaluation and testing endeavors. This 
sumaary was prepared *froa intra-Pi^o ject discussions, was 
entirely narrativp, and did not include' illustrative 
vignettes or quotations^ 

The remaining^ working papers presented thematic 
analyses of the ' interviews. Recurrent themes were analyzed 
in the, working papers and illustrated by ^otations^ from 
the written interview suaaaries. * 
Descriptive ^ ecu rac y # s 

There was little opportunity tq independently verify 



tht accuracy of ths' interviewee* s remarks. The interviews 
were relatively brief. The three interviewees within a 
school did not all discuss the saae events or contextual 
conditions, reducing the opportunity for triangulation. 
Ana there were no follow-up inter vie*s~-in which to pro be , 
clarify, or cross-exaaine. As noted in the aethods 
description, however, fifteen sampled interviewees were 
given the chance to consent on the written suaaaries of 
their interviews. This provided a check on the summaries 1 
adequacy but did not insure that the interviewees' 
descriptions and interpretations were accurate. 

Accuracy, in this situation, is best examined in the 
aggregate. Any-descrrptlun o r opinion which recurred in 
several interviews was aore likely have soae factual 
basis. Behavioral patterns, visible in the interviewees 1 
descriptions of the way they made -decisions in their 
schools, also seeaed worthy of confideirbe. 

There was a clear effort to check the data for 
consistency and plausibility. The analytical, theaes were 
disqussed at le-ngth, too, and efforts were aade to search 
*out supportive and nonsupportive data. The'case experts 
aajor responsibility was to serve as stringent critics 
ba^pd upon data in the interviews they had been assigned. «. 
H^ture of the Analysis 

As already Mentioned, the analysis sought to 
generalize froa the interviews, identifying recurrent 

8 8 * 0^ * # t 



opinions and behavior patterns, or broadly applicable 
interpretive themes. Pour thematic analyses were written* 
One paper differentiated evaluation activities into three 
classes: formal evaluation, informal evaluation, and 
evaluation-like activities. Exact definitions of the 
classes were not given, but major exemplars of each class 
were enumerated* The paper's tripartite classification of 
these activities was "etic," i.e., imposed by the analyst 
and not indigenous to the respondent grpup. 

A second paper summarized the reported attitudes 
toward, and uses of, several specific evaluation 
activities: state program reviews, needs assessments, 
standardized testing, etc. Information relevant to this 
paper was available in most of the interview summarips. 

Another paper examiJ^d^the interviewees 1 explanations 
for evaluation's influence (or lack of influence) in their 
schools, along with interviewees 1 suggestions for 
evaluation improvement. The author argued that three 
categories of factors recurred in the interview comments: 

Icimity, competing demants on time, and psychosocial 
relationships. These categories were explicated and 
further subdivided.. The implications of the- subcategories 
for evaluation utilization were discussed. 

The fourth analytical p^per proposed that school ' 
principals' management styles could be differentiated.' Two 
aajor styles were x d iscerned aiong the^ interviewed 
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rincipals, ana it was argued that the styles were 
associated with different patterns of evaluation use. The 
styles were not defined unambiguously , however* 
Gene raliz ability 

Sample size and representativeness were more 
satisf^s&ory in this study than in the case studies or th£ 
Evaluator Field Study. Twenty-two ESSA Title I funded 
schools were sampled randomly fz6m within fletro school 
district, and sixty-five interviews were conducted. We 
believe that Interview Survey adequately captured the 
attitudes and perceptions of Hetro Title I personnel. 

Generalizability beyond the Hetro population - 
cannotserted be asserted rigorously. Hany of the issues 
identified in the interviews have fceen mentioned by other 
school researchers, however, suggesting that the findings 
were not inherently idiosyncratic to Hetro. 
Points of Concern 

I ntervi^w Hete rogeneity. Pecent reanalySis of th 4 e 
interview tapes has indicated that, across* the 65 
int er vie ws, there was greater heterogeneity in topic ~ 
coverage and eaphasis^than had been anticipated. Despite- 
expensive training, interviewees did not always elicit 
appropriate information", arid s6me interviewers 
systematically diverged from the intended topic emphases.* 
As a consequence, missing or incomplete data was not an 
uncommon problem. It may be that this heterogeneity is 
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almost inevitable when a topic-guide interview i£ 
administered by several different interviewers* To correct 
the problem, . one could either change to a standardized 
interview (i.e., one with «a fixed question script) or 
reduce the number of interviewers. Two or three highly 
skilled interviewers, conferitfg frequently, might hold~\o a 
common interview format more successfully, in future 
studies we would be inclined to select cne of these' 
options. If a panel of several interviewers was necessary, 
standardized interviewing would seem preferable. 

iack of Quanti fi cation . The thematic analyses 
suffered from a lack of tabulations or cro&s-tabulations of 
relevant evidence. For example, one analysis asserted that 
"competing demands upon time" were^ften cited as 
discouraging evaluation utilization, but no indication was 
given of how many interviewees mentioned this ^actefc. 
Another paper related two different management styles to 
different patterns of evaluation. This relationship could 
have been illuminated by an appropriate cross- tabulation, 
but w^s not. - *• 

Two circumstances discpuraged such quantitative 
•.'displays. First, on any given topic, many interviews were 
incoftclusi ve. 'To take one of the preceding examples, many 



interviews did. not touch up6n the question pf time dem 



. ^ie^canno*t know* if these .interviewees tf^uld have agr4ed 
that competition for time was a significant impediment to 
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evaluation influence. Nevertheless, a tally of the^number. 
of interviews which mentioned this factor might have been' 
informative r- except for a second difficulty , the lack of 
fall tape transcripts* A tally based sol&ly upon the 

* interview summaries could^easily be in*error. only by 
examining 6 the interview tapes or a full written transcript 
could one accurately determine how to tally an interview** 

Lack of Full Tra nscription *" In general , the lack 
of. full interview transcriptions was troublesome* It 
impeded quantitative analysis, as indicated above* * 
- moreover, the typicality of the few quotations which were 
noted in the summaries (often because of their pithiness or 
color) may have been overstated* Finally, if transcripts 
had been avfilable on a timely .basis, senior Project staff 
might have befen able to monitor interviewer performance 
more readily* 

| Transcription is an^j^rous burden, however. Other 

researchers have reporte^^n average four to one ratio of 
transcription time to tape time (Patton, 1980). Full 
transcripts would be a superior data analysis resource, .but 
budgeting for transcription is* difficult. 

• " Occurrences " versus " Decisions ** In the 
interviews, respondents >werp asked discuss two or three 
"significant * occurrences"^ in the rfecent history of tfce 

* 

program* The way these occurrences — program changes, 



decisions,- new acti vj,ties* new requirements, etc. — were 

f r ^ , » ; - ■ 
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handled bj the school was investigated in some detail* 
v , In hindsight, we would have done bettet to Vsk the 

4 

interviewees to describe recent significant decisions . 
That change in wording would have focused interviewers 1 an* 
interviewees' attention on the decision making and fanning 
activities associated with prograa change, tppics whi^ 
were^ our maj or interest. Unfortunately, the interviews 
often dwelled, instead, on the substance of the change s ? and 
on how arrangements wer<e made to implement new program\ 
activities. The n&ed'for this .different -wording might Ifave^ 
been apparent if there had been a ^05^ extensive pilot test 
of the interviews, a test in which all the prospective 
interviewers had participated. 

Subjectivity. The topic-guide interviews gay^Tthe 
interviewer considerable latitude, which we know affected 
topic coverage, one mast wonder whether differences in 
Question phrasing and in -responsiveness to interviewee , 1 
remarks could have subtly cued the interviewees and 
affected not only topic coverage but also the tone and 
^content* of the interviewees 1 responses. * 

% Summary B eview 

-* * 

Th& Interview Survey genuinely seeemed to evoked 
candid, interesting data from interviewees. During the 
hour long interview, the intervie we«*s t appeared "to become 
accustomed to the tape recording and seamed to speak 
honestly, and soictines quite insightfully, about their 
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schools, at the conclusion of the study, we felt that much 
had been learned about school decis'ionaaX^ng and about the 
role oje evaluation at the school level. Moreover, the 
number of interviews undertaken, and the sampling 

0 

procedures, engendered confidence that -the findings would 
be representative -of Metro Title I school perceptions. 

let as more rigorous and aore guantitative analyses 
were undertaken, another side to the research experience 

* 

eaerged. It was extreaely difficult to rigorously define 
teras or guahtify the data. Holistic analysis was 
possible, and general analytic theaes cculd be identified 
aid communicated through judiciously chosen illustrltions 

I 

and quotations. But -aore systematic analyses / 'were 
frustrated by the slipperiness oi the concepts involved, by 
the imprecision of the- language used by interviewee's (and 
interviewers, as well), and by th4 diversity in coverage 
froa interview to interview. The survey saapling design 
had held out the promise of useful, straightforward 
statistical analyses. The reality was aore complex, aore 
difficult, and less amenable to quantitative reduction and 
analysis.* 

> 

- Also, the use of the topic-guide interview approach 
was, instructive. Even tfhough the interviewers had. had 
several weeks of £roup -training,^ and despite the use of 
validators and standardized data summary forms, there was 
substantial variability ib technique and emphasis from 



interviewer to interviewer* Perhaps signif icantly, *a brief 

pilot test was not sufficient to. re veal this potential foil 

i 

variability. * j 

In all, the Interview survey convinced us of the power 
of interviewing, and also of th^ sizeable challenge in , > 
guiding such studies to a successful end# Fortunately, we 
. wi shed ,t_o_ make an exploratory investigation of behavior and: 
attitudes in Hetro, and the research results set our needs 
quite well # If more stringent hypothesis verification had 
been the goal, instead, then the survey experience would 
have been less satisfactory, 

/ . 
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Chapter 5 
Epilogue 

ill the research efforts described in" this report 
contributed significantly to our understanding of 
evaluation information use in local schools* Heasur^d at 



this bottom line, all the Methodologies have been 
successful. The methods have their differences, however, 
in the demands they place'upon the researcher and in the 
uses to which they are aost appropriately put. 

The Evaluation case Studies and the Evaluator Field 
, Study employe^ methods which were almost complementary 
faces of the s^ae coin. Both efforts' were case studies, in 
fact, because the Evaluator Field Study was actually a case 
study of the evaluators* work. Tke studies differed in 
their primary mode of data collection, of course. The Case 
Studies relied heavily on interviewing; vhe^feas the Field 
Study favored observation and on-the-spot, informal 1 
conversation. Both studies generated* narrative datanotes. \ 

The deficiencies of one study were the strengths of 
the other. The interviews in the Evaluation case studies 
# gave us hearsay data^but from several strategically placed 
informants. The Field Study yielded compelling, first hand 
.observations, but observations anchored to a narrow vantage 
point. 

neither interview case studies nor participant 
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observation reguire that the researcher have a strong pxior 
understanding of the phenomenon under study* So both are 
suited to exploratory research* One cwr learn as one goes, 
although an awareness of alternative .theories or conceptual 
frameworks can be invaluable els one tries to aake sense of 
the data* 

The, Case studies and the Field Study coaplemented each 
other so well that an a^algaa of the two would probably 
have been superior for our research purposes'. The' case* 

studies would have benefitted if the researcher had had the 

t 

opportunity for a brief iaaersion in the culture 1 of the 
schools studied* The field study could have profit ted from 
a wider set of informants prorbably as interviewees* It 
se^as foolish to segregate observations fro* interviews, 
although teaa research aay be necessary to effectively 
reach distinct informant groups* The stimulation afforded 
by teaa research could also help to counter the ^ 
routinization mentioned as a hazard in solo fieldwork. 

Both case studies and participant observation , 
typically focus on just a few cases or settings, asking 
generalizajjil^ty a concern* Findings from saall samples 
can generalize, of course but the problem is convincing .a 
skeptic (or soaetiaes oneself) that the findings will do 



so« 



* « 

The user Interview Survey, in contrast, yielded 65 
<>ne-hour interviews at 22 randoaly selected schools. But 
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sanfple size and sample^ selection method were only the most 
obvious dimensions , on ^which the Interview Survey differed 
from the other studies. Interview planning, instrument 

development, and interviewer training were ail very 

» * 

demanding, requiring much- more preparation and lead-in time 

than had the other studies* The select ion of a tO£i.c-gjiide 

strategy eased^the instrument development task somewhat, 

but it increased -interview variability* consequently, 'it 

was more difficult to carry out quantitative analyses which 

* 

could exploit the aethod f s inherent sampling advantages. 

The theaatic, non- quantitative approach taken in the 
Interview survey analysis has substantial precedent (e.g. 
David, 1978; Kennedy et al. , 1980). We were satisfied that 
useful results had emerged. Procedures for theaatic 
analyses are not codified, however, ai\d one hesitates to 
make the same claims for the genera lizability of* an 
analytical theme thafc one might make 'for the 
generalizability of a quantitative result from a sample of 
similar size* We believe that more attention should be 
given to the problem of analyziirfj this type of interview 
surveys 

All three methodologies have*, as we hfcve said, 
conttibuted to our understanding of evaluation information^ 
u^e ift the schools* Hone of the methods is clearly 
superior; all have advantages and disadvantages* Just as 
clearly, researcher judgment aust be exercised at aany 
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points throughout the studies. Tn the research- 
descriptions and the critiques, we have discussed the 
decision talcing which shaped the research, and the 
raaifications of the" decisions aade. In hindsight, there 
are aethodological decisions which we light aake 
differently today. Birt nxone of tfa^ choictes which were aade 
were disastrous to data guality or validity; their effect* 
was actually ignite modest, on balance, Me believe that we 
hare been well served by our gualitative explorations of 
school evaluation activity. Proa r<?assessaen ts such as s 
this one, we hope to be able to design even better studies 
in the years to coae. 
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FRAMEWORK FOR* ANALYZING EVALUATION SITUATIONS* 



Category 1: Preexisting Evaluation Bounds c 

Property 1.1 School community conditions 

Propetty-1^2 Mandat^4>etmd5-^f-a^^ 

, Property 1.3 Fiscal constraints 

Property 1.4 Other nonnegotiable requirements 

Category 2: Orientation of the Users 

Property 2.1 Questions or concerns about the program 
Property 2.2 Expectations for the evaluation 
' Property 2.3 Preferred forms of information 

Category 3; Evaluator's Approach 



Prt>per£y 3.1 

Prop erty 3.2 

Property 3.3 
Property JL4 

Property 3.5 

Property 3.6 

Property 3.7 



Use of a fomfel evaluation model 
Research and analysis considerations - 
"Choice of role 
User Involvement 

'Dealing with mandated evaluation tasks 
Rapport 

Facilitate and stimulate the use 
of information 



Category 4: Eva! uator Credi bi 1 i ty 

Property 4.1 * Specificity 
Property 4.2 Changeability 

Category 5: Organizational Factors 



Property 5.1 



Property 
Property 
Property 
Property 
Property 



5.6 



Interrelationship between site 
and district 

Site-level organizational arrangemepts 

?ther infprmatton sources 
eacher and staff views 
Student views ♦ - 
Costs and rewards 



.Category 6: 



€xtraorganizational Factors 

Property 6.1 Community influence 

Property 6.2 Influence of other governmental agencies 

\ - 



* ~ *fromAlk1n» M.C., DaTllak, R. $ & White, P. Using Evaluations: 

Does Evaluation Make a Difference? Beverly HIJls, .California: 
^ Sage Publications, 1979. 
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FRAMEWURK FOR* ANALYZING EVALUATION SITUATIONS* 



( 



Category 1: Preexisting Evaluation Bounds 



Property 1.1 
Propel U2 
Property 1.3 
Property 1.4 



School community conditions 

Mandated-beunds^f-an-^yait^^ 

Fiscal constraints 

Other nonnegotiable requirements 



Category 2: Orientation of the Users 



Property 2.1 Questions or concerns about the program 
Property 2.2 Expectations for the evaluation 
Property 2.3 Preferred forms of information 



Category 3: Evaluator's Approach 



Use of a fomtl evaluation model 
Research and analysis considerations - 
Choice of role 
User Involvement 
Property 3.5 'Dealing with mandated evaluation tasks 
Property 3.6 Rapport 

Facilitate and stimulate the use 
of information 

/ 



Prbperfy 3.1 
Property 3.2 
Property 3.3 
Property JL4 



Property 3.7 



Category 4: Eva 1 uator Credi bi li ty 

Property 4.1 * Specificity 
Property 4.2 Changeability 

Category 5: Organizational Factors 



Property 5.1 



.Category 6: 



Property 5.2 
Property 5.3 
Property 5.4 
Property 5.5 
Property 5.6 

»Extraorganizational Factors 



Interrelationship between site 
and district 

S^te-level organizational arrangemepts 

?ther infprmation sources 
eacher and staff views 
Student views ♦ ■ * 

Costs and rewards 



Property 6.1 
Property 6.2 



Community influence 

Influence of other governmental agencies 



* from Alkin, M.C., DaTllak, R., & tihite, P. Using Evaluations: 
Does Evaluation Make a Difference? Beverly HIJls, .California: 
Sage Publications, 1979. 
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Category 7: Information Content and Reporting 

» Property 7.1 Substance 
\ Property 7.2 Format 

Property 7.3 Information dialogue 



, Category 8: Administrator Style 

^ Property 8.1 Administrative and organizational skill 
4 f\ Property 8.2 Initiative 
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OBSERVATION FOCI 



1.0/ 

2.0 
2.1 
2.*2 
2.3 
2.4 
2.5 
2.6 
2.7 

.3.0 
3.1 
3.2 
3'. 3 
3^4 
3.5 

4.0 
4.1 
4. -2 
4.3 



„ 5. 
' 5. 

5. 
.5, 

5. 



5, 
5. 
5, 
5, 
5. 
5. 



6 
7 
8 
9 

lb 
li 



Miscellaneous Evaluation and Testing D-uties 
*• ' i 

Misc. Comp. "Ed. Duties -f 
Liaison Visits 

Testing- Services • , * 

Ongoing Planning and Evaluation 
Needs assessment i 
. School plans 
Mock reviews 

State Program Quality Reviews 

Child Service Program Duties m . 

Liaison Visits 

State preschool Evaluation 

MH/SS Survey 

Contextual constraints ' 

Other data collection activities , 

PLPSS Duties * 
£upil testing ' , * • ' 

Other data collection activities ^ 
Evaluating sta^f meetings 

Evaluation Work Tasks ^ 
Learning one's * job 
Attfencfdng staff meetings 

Planninipand designing evaluation efforts 
" Preparation activities, - • 
Interactions # 

5.5.1 Exchanging* information 

5.5.2 Making recommendations 

5 1,5. 3 Building relationships v 4 
Collecting evaluation data 
1 Analyzing evaluation data 
^Reporting, formally dr informally 
Giving public presentations 
Representing the E & T Office ■ s , 
Giving misc. technical assistance 
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6.0 Special Topics % m 1 

*6.i Attitudes towards, evaluation and testing 

6.2 Attitudes towards the 'evaluator • * 

6.3 * Bilingual education issues ' . 

6.4 CWifi^entfiality ' 

6.5 Situations 'marked by conflict" • # 

6.6 Evaluator's follow up, activities 

6.7 / The role and functions of "evaluation * T 

6.8 Who initiates evaluation work? , 

6.9 Who is involved? Who participates? 4 

6.10 Evaluation methodology , 

6.11 Organizational constraints • - 

6.12 'Regulations, Requirements, and Policies j 

6.13 m Participants 1 roles 
6\l3.1 Client roles . • 

Jd.13.2 Evaluator roles 

6 ; .1C r .(Open) 

6.15 Sensitivity i to clients* ' ■ 

*6.16 TJxe use pf forms, recipes,* and other formal structures 

6.17 Perceptions of evaluation success 

6.18 Supervisors and supervision 

6.19 The role of testing in evaluation 
^-20 Timing and time constraints 

^ Interpersonal relationships , 

T5.22 "Compliance" issues* 

6.23 Especially significant individuals * 

6.24 Use of memoranda ; 

6.25 Efforts to build* evaluation capacity * - 

6.26 Parents and the community in evaluation \ 

6.27 n The Schools 1 Point of View" 

6.28* The effects of positive vs. negative findings 

6.29 Special evaluator characteristics ^ 

7.0 Fieldwork issues 

7.1 Rapport ' 

7.2 Fieldworker 1 s asides 
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lame of' Interviewer: [ School. Code:__;_ 

Respondent. Code: 
INTERVIEW GUIDE • > Title: / f' ' ' 1 

6 

). Introduction 

Who v/e are in televiewing , . . " 

. Why ("uses of Information in special programs 11 ) 
Confidentiality . • \ 

Appreciation * 

1. Description of Specially-Funded Programs 

„' (Consolidated Project) ' 

4 _ 

2. Du^s & Responsibilities > s 

3. "Significant Occurrences in the Life of the Program" 
Changes (personnel, goals, materials, attitudes, eic.) 
Rejected Alternatives • 

"4. Factors Affecting Identified Occurrences 

Description/History \ • 

Different Influences „ 

Resolution Process ^ 

*> , 

5. Role of Evaluation in Identified Occurrences 

• - 

% 

6. Role of Evaluation in General 
- Administrative Level 

(Witirin- school, District sponsored, PQR & N v * 

Bmock review) * ' 

Description 
] Influence on Action &* 'Attitudes 

4 , Factors Affecting Impact 4 • 

v Improvement? 

(Repeats appropriate: 5. Role of Evaluation in Identified Occurrence) 



•7. Additional Comments ' 
0 4/25/80. V Mr, 
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Interview Topic Description * 
(Training Document) 

a 

' Introduction to the Study 

The purpose of this research is to determine the role that 
information, particularly evaluation information, pl^ys in school 
level pro.gram decisions. It is difficult to ascertain the relative 
importance of evaluation information directly,. Asking about 
evaluation tends to bias the respondents' recollections towards 
„ just those situations in which they did consider information 

from evaluations. Instead, the school-level decision makers will 
be asked to identify significant occurrences in the life of the 
school programs. The situations they select will be analyzed to 
.determine the 5 f accrrs that affected their beliefs and' actions. 
Amour these fac-irs itey be evaluation. 

Hour long interviews will be conducted with school-level 
administrators, who might be users of evaluation information* 
These will not be structured interviews with rigid protocols, but ' 
naturally evolving conversations gjlided toward certain carefully 
selected^ topics. The topic guide is outlined below. The precise 
wording of questions asked by each interviewer will not be pre- 
determined, rather it will evolve within the topic framwork as 
part of the, natural conversational style o£ the interviewer. 
Similarly, the exact ordering of questions will be an inter- 
active function of many factors, including, for example, the 
focused or. diffuse quality of the respondent's answers, etc. 
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Moclel Introductory Remarks - + 

• Hello, my name is _ m we arc interviewing 

elementary school administrators to investigate the ways they 
use different types of information in school planning and ad- 
ministration. We are particularly interested in schools with 

specially-funded, supplemental programs. 
* ■ 

I can assure you that everything we say in this interview 
.1 be strictly confidential, and any reports that are written 
will be completely anonymous. If you do not object, I would 
like to tape record our conversation. It allDws me to capture 
your thoughts correctly, c^id makes our work much more accurate. . 
However, if at any time you would like to stop the recording for 

moment, please indicate that to me and I will turn off t? 
machine. 

m 

* 

I would like zz start by asking you for a brief description 
of the specially- funded programs here at school. 



Description for Interviewers . - 

Topic Area 1: Specially-Funded Programs in the School 

A basic knowledge of the nature and scope of the specially 
funded programs in each school is necessary to understand the con- 
text in which decisions occurred/ Initially , only a very general 
description will be 'sought; specific details will be elaborated 
as part of the subsequent inquiry into selected events and ' 
occurrences . 9 

Model Opening Question: I think the easiest place to begin 

is with a description of the program here at 

school. Can you give me a very brief description of the 

programs you ha#> here as part of the* school's Consolidated 

Project? 

Topic Area 2: User's Position and Responsibilities in the School 

We also ne^i to know each respondents duties and responsibilities 
in the school. Z~ particular their administrative relationship 
to the school *s special programs* will be important. At the out- 
set a very general description will suffice. Details will be * 
obtained as specific decisions are investigated later in the r 
interview. 

Model Opening Question: Can you give me a general descrip- 
tion of your job and what your duties are with respect 
to the programs you just ^^scribed? 
Topic Area 3: Significant Occurrences in the Life of the Program 
This is a crucial question, for the respondents answer will 
determine the situations on which the bulk of the interview will 
focus. Ideally, each respondent will be able to recall signifi- 
cant program decision s in which they participated. Realistically, 
however, the evolution of a school program is more a matter of 

" lit; 



incremental change than formal "decision" events . Thus, each 
school* administrator will be asked , to identify two or three 
events that they believe were "significant occurrences in 
the life off the program(s) " . Subsequently f the interview will 
focus on these occurrences and the factors that affected the 
described outcomes. * 

Model Opening Question: As is said at the beginning we're 
interested in the way information is used by school admin- 
istrators. To talk about this I want to identify 2 .or 3 
particular situations. I would like you to think back pver 
the past two year:-; and try to reckuL two or three signif - 
icant occurrences in the life of tqe program here at 

school. I realize that this question 

is somewhat vsgue, but it is vague on purpose. I' want to 
get your ir.cression of what was importnat rather than mine. 
_ Try ar.d recall a few different occurrences that you thought 
were significant in determining the shape and character o£ 
the program during the last two years. For now I*d just 
like to list two or three such occurrences. We'll discus^ 
the details later, 
sub topics: 

— changes (personnel, organization, goals, curriculum, materials/ 
activities, attitudes, other milestones, etc.) 

4 

— rejected alternatives 

— reinforcements in points of views, attitudes 

✓ 

Topic ?^rea 4: Factors Affecting t he Specified Occurrences 

'To determine the relative .contribution of evaluation infor- 
mation in the total decision context, the respondents will be 
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'was not mentioned, thc*qucsti on will bo specifically asked by the 
intervifcv;et at this stage in the interview. 

Model Opening Question: Did evaluation make ahy difference 

in* this situation? ' 
Topic Area 6: The Role of Evaluation in General 

To this point, evaluation has appeared as a secondary con- 
sideration in the interview. % The situations identified by the 
respondents were allowed to define the scope of the discussion. 
Now, evaluation will be considered in its own right, and the 
respondent's, wider knowledge and contact with evaluation will 
be investigated. • * 

Model Opening Question: We've discussed 

4 — 

and [ in great detail { and I think 

I understand -he important factors involved in those occurr- 
ences (brief Elaboration)., Dr. Alkin and I are particularly 
interested the usefulness of information from evaluations 
I r d like to =sk you to shift your thinking from these 
specific situations to thinking about .evaluation in general. 

V / 

Will you take a minute to recall the program- evaluations 
that have gone on in the pkst year or two; .then, try to 
tell me what impact they had on you and on the programs at ' 
the school? 
sub topics: — 
— level (within school, districb sponsored activities, PQR and 
"mock review") 

— characteristics of the evaluation (formal/informal, content, 

style, personalities, method of communication, etc.) 
— its influence (on actions, attitudes, etc.)' 
—improving evaluation usefulness 
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Topic Area 5: The Role of Ev alu ation in the Identified Situation 

After the more extensive discussion of evaluation it may be 

t 

appropriate to repeat the earlier inquiry into significant 
occurrences. Certain sfabtle evaluation influences may have 
emerged froin^the lengthier discussion which were overlooked 
|* previously: 
c Topic Area 7: Additional Comments 

fc At the conclusion of the interview, there will be a brief 

| open-ended discussion period. Respondents will be given the 

opportunity to modify or expand their previous comments and 
|T .clltirfy any misinterpretations. 

| Model Opening Sentence: Before we ponclude, I want to give ^ 

you an opportunity to make any additional comments about 

r | our discus = Is there anything you feel should be. 

I 

clarified :r expanded with respect to/the situations you 
I identified, the various factors you singled out or about ) 

r evalua-ior: in general? ^ 



i 



Thank you very much for your cooperation. 
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' User Survey , - ^ 

Mock Interview (Questions Only) • * 

"Introductory Statement" .•••let's begip" 

"A simple place to start would be for you to give me general description 
of the special prpgrams operating here." * 

• "What I mean is your Consolidated Project. I imagine you are ^receiving 
funds from a number of different sources — maybe Title I, or Miller- 
Unruh; School Improvement, whatever. lid like you to tell me which 
programs are operating here and* briefly describe what you are doing with 
the funds." ' — I 

I see. You have Title I funds, School Improvement, and Title VII. 
Describe the Title I project for me briefly? 11 

»• 

"OK. What about Title VII?" ^ 
"And how are you using the School Improvement money?" 

"Thank you. I think I. have a general picture, but let* me double check. 
I'll descril?e your project to you, and you tell me any ways in which the 
portrayal isrtSnaccurate : , f The Title I funds are used.;... 1 (brief 
restatements of project description)" 

"I want to talk about some of these program areas in greater details later, 
but first I'd like to know more about your particular role at the school. 
Can you give me a -brief description of your job?V 

<"OK, and what are yo.ur responsibilities as far a$ the special programs 
are concerned?" 

"I see, in general you have supervisorial duties/ for all three projects? 
Tell me, do you separate them in your mind, or dp you act towards them 
as if they were just one unified project?" / 

"Which makes the most sense for me, to discuss them separately — first 
Title I, then Title VII, etc. — or to .ask abo^t the project as a whole?" 

"Fine. Befare we go ion let me try to restate what you said. Your 
primary^ responsibility, the thing that takes up* most of ytfur time, is 
supervising and planning for the project." v 

"Thank you. I appreciate the correction. I don't want to oversimplify. 
Besides planning and supervising, what other kinds of activities are , 
you called on to perform?" 

"A sort* of jack-of-all trades, then?" 

"OK, I think I have enough of an overview to start. As we proceed I may 
ask you to clarify aertain things for me. The next question is probably 
.the most difficult one in the interview, because it is- somewhat vague-. * 

• Not only that, but I'm going to ask you to review the past two years of the 
project in your mijid and pick out certain things for me. I would like you 
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t<5 think "back .over the last two years and, from your administrative per- 
spective, identify two or thtfee significant occurrences 'in the life of*^ 
the program. Latter ^p'll talk about each, of jthem in detail. Right how 
I just want a list of. two qr three occurrences that you thought w$r<p 
significant in deciding the shape and character of the project" 

* ** i 

"OK, the State's decision to award you School Improvement funds was cer- 
tainly significant. I guess I didn't make my question clear enbtigh, be^ 
cause I want to know about events or deliberations or changes that hap- / 
pened" within the school's discretion, not outside your control. Can you ' 
recall any of this type of occurrence and'tell me briefly a£out i£?" 

'"I realize that it's not exactly clear, but I'm being vague on purpose. 
I don't want m£ point of viejv about what is important to determine your 
answer. I f d want to learn your definition of what things were important. 
But I can give you some .example of the type of thiftg that might have 
occurred. " t < ' % > 

"I want to know about important changes or modif icatdons of program 
actions that you took here at the school during the last couple of years. 
They could be part of developing the project plan, they might be per- 
sonnel changes, or modification in the project approach, pr new materials, 
or things <^f that nature. If you tl^ink .about the school program as it 
^is now, I waiqj: to know the significant occurrences over the last two 
yea^s that determine^ where you are now." m * 

• „ * 
"That's what I'had in mind. When y6u decided to reduce the .number of 
resource personnel and hire^more classroom aides. That's a perfect ex- 
ample of what I'm interested in. Can you recall one or two more such 
actions that you, think were important in the shape of * the project?" 

"Good, you purchased a new set of Individualized math worksheets and 
student record slteets. Can you think of one more example? It may not 
actually have been a change, but something you considered and then re- 
jected.' m A decisio'n not to do something." 



"OK, if you don't recall anything else right now, the 
be excellent. Later on, if something else occurs to 
know. " 

"I want to discuss each of these events in more detai 
to the first event you identified. What I would like 
tell me how it came about that you decided to reduce 
and hire more aides. I want to know all the factors 
this decision, both pro and con. Give me a much more 
tion of what happend." 



se two examples will 
you, v please let me 



1. Let's go back 

you to do is to 
resource teachers 
that influenced 
detailed descrip- 



"OK. It was at the time, you were planning the annual project applica- 
j tion for last year." 

■iWheh was that?" - . 

"And what happened >" . 



I see. 



"When you say that the teachers wanted more aides, how did you know that? 
I'm not just trying to be dif ficult , : btat what I am particularly interested 
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1 in is how different people's ideas and wishes were communicated? How 
was the desire to have aides made known?" . . H 

♦ 9 

"Let me go back a minute, who sat on the committee that drafted the new 
application?" * 

"So, it was a three "member committee with lots of large subcommittees, 
one for each area, *In which part did the decision to go with morec 
* aides- arise?"- * 

i 

"If it had been talked about v f or a long time and mentioned at faculty 
meetings in the past, what were the reasons why people felt this would 
. be a good idea?" 1 4 

^"The teachers at the school were pretty convinced that it would help 
them do a better job. " 

"They said that dealing with 'all the IEP's had really become a headache. 
Would it be faiay to say that the presence o£ all the LES/NES kids ac- 
celerated the need for more assistance with paperwork and- small groups?" 

J^hear you saying that there was a sort of general groundswell for more 
^axde^, and that paperwork was becoming impossible, as was classroom 

management, with all the LES/NES kids. Was there anything else that 
' influenced this decision?" 

"Well, for example, what did people think about the job the resource 
teachers were doing?"* *' 



"I see. There was 
people d±dn,'t care 



some support for the resource te^phsis, but a lot of g 

that much?' What was going to happen .to them?" 

■» - » 

"Oh, it wasn't really a question of firing them. How important was'l£ 
that everyone knew there would be classrooms for them this year, and 
they wouldn't hav£ to 'transf er> to new schools?" > , , 

"Were there any outside, influences?" * . ^ " ' ' 

"By that I mean, at the area level the district Ihvjel were there sug- 
gestions to. gd with, more aides?" ' * . 

".But everyone knfew that people were' doing it a lot?" 

"You mean it Was sprt of a trend in the district and everyone wante<J aides 
"How it ,make^|^aore seijse to me." * ' 

"Let me try to summarize and you correct me: There was sort of a ground- 
swell for aides in other schools, and teachers were really feeling the 
increasing pressure of classroom management taska^with all the IEP and 
stuff. The resource teachers were liked but the§fc jobs were dispensible 
and there wasn't too* great a personal loss sjLnce they could all still 
work here, so there was sort of universal desire for more aides* How 
would you modify th^rt summary?" x 

,, RighJtf' ,, * ' 



r 
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* 

"One last thing. This seems to have been based on common knowledge and 
'cdmni^n sense* Was there any evidence that aides would make a difference?" 

"What it boiled dpwn to then was you knew you had to. do something be- 
cause the test scores were low, and aides seemed to fit the problem as 
you saw it. 11 

"Fine, this has been very interesting* I appreciate all the detail you 
were able to recall. Now let's talk about the other situation for 
awhile — when the school decided to switch from your old math program to 
the individualized student inventory program, PRISM. This mast have 
been a more complicated decision. Tell me about it." 

"Uh huh." • • • 

"I see. Do they have these demonstrations very often?" 

"Oh." / * * 

* / 
"L£t me try andwreyiew these points one by one in*my own words. First, 
your math test scores were^poor. How did you know that?" ' 

"When did you find out each year? 11 

"How bad were they?" . 0 

"Does the district give you any advice, or do you just get the numbers?" 

"Do^g the evaluator give you any suggestions or just explain thqX^est 
scores?". * . 

"OK, so you knew you had to do something, and you saw PRISM demonstrated 
at the CMC conference. When you suggested it, ^id everyane automatically 
go along with your suggestion?'! 

"How did you present it^to the teachers?" 

"You made a case for PRISM. What ideas did other people have?" 
"What's-HappyMath?" , . 

"Let me interr^pt^ f or a second. I don't really need an explanation in 
that much detail. I'm impressed that you remember this program in such 
careful detail, however. Let's get back to the actual choice. Weren't 
there any advocates for the old program?*/ 

"What were their reasons?" * . 

"How did you decidfe?" * 

"What did people* base their votes on?" 

"So PR]^M had test results^ and HappyMath didn't? V?ere there ahy other 
inputs into the choice?" 

"Is it true that you had the power to make this decision here at the 
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school without 'any comment from the area staff, or downtown or from the 
state?" 

"The district was pushing more individualized programs, but did thev 
suggest PRISM in particular?" 

♦ 

*In what way were they "pushing" for more individualized programs?" 

"Individualization seems to be the big thing. Was HappyMath individualized? 

"So it was going to be one or the other, "the old program was pretty much 
doomed by the times?" . , 

"This whole process is very interesting to me. Were there any other' 
factors involved in shaping people's opinions before they actually voted?" 

"Were the results from PRISM at other schools good?" 

"But the teachers at other schools liked it? How did that word get around?" 
^How many teachers do you think went along just because you recommended 

1 1 ? 

f 

"Realistically how influential do you think your particular point of 
view is?" 

s V 

jTake a minute to think. Was there anything else that -entered into the 
final decision?" 

"The PTA committee preferred PRISM. How much did that matter?* If they 
had preferred HappyMath would it have changed the decision?" 

"Anything else?" 

"Let me try and summarise. Please tell me if I have left anything out: 
Your test scores were low and you knew you* had to do something. Though 
the district wasn't suggesting any program in "particular, there was a 
push toward individualization etc." » * 



Good. I don't want to forget the fact that the teacfie« got to review 
the materials at the faculty meeting. And, though it was the math sub- 
committee that made the final recommendatibn, almost everyone really 
.had an input.", ■ . - 

t 

"This has been very useful. Of all tfee things that went into this situa- 
tion, which influences would you say were the most important?" 

"The direct. contact was the key in your mind?" 

"Before* we l^ave these two situations- I want to ask about one thing in 
particular — evaluation. In the Math program case you told me ,that the 
district test scares really motivated you to make the change. Did 
evaluation or the evaluators influence that decision in any other wa^?" 

"SO, Mr. Bertrand confirmed your enthusiasm for PRISM. That at least 
told you you ^ere on safe ground. Did \ie talk to the staff?" 

< L 
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"Then his influence was only indirect , and not too great?" 

"What about the classroom aide decision. In what way did evaluation 
influence* that situation?" ^/ 

"I didn't think so, but I wanted to ask you specifically. Essentially 
there was no hard data or evidence* for aides one way or the other, nor 
for resource teachers." 

"I really appreciate the care with which you are approaching my questions 
For fehfe rest of the interview I'd like to change focus a bit. Is there 
anything else you would like to ..add before I switch topics?" 




M 0f particular interest to me is the role'^hat evaluation plays i*i 
school program improvement ar\d change. At this point I'd like yau to 
think specifically about evaluation, rfot just in the two situations we 
discussed, but in any way; that it affects the school. . I'm going to ask 
you to think back over the same two year period and recall all the dif- 
v ferent: contacts- you've had with evaluation and evaluators. Then I'm 
going to ask you something about these contact?. " 

"First can you summarize for .me the ways. if any you come into contact 
with evaluation?" 

You get a regular visit every couple of months from the evaluator. 
% 4 Does that iiave to do specifically "with the Title -I program or with the 
whole Consolidated Project?" 

"B.esides^jtbat visit, *how often do you see test scores? 
' "Are/these contacts helpful to you?" 4 
"In what ways do they influence your decisions?" 
"In what ways do they influence the program?" 

"What is itabout the evaluation that makes these visits so useful?" ' 

"I want to structure^ this discussion a bit by ^asking you about three dif- 
ferent levels of evaluation activity, and way^ to improve each one. 
First, I'd like to know if/ there is any evaluation that goes on at the 
m school itself, not district level or area level information, but things 
you might do yourselves that you cbnsider to be evaluation?" • 

V "Because your budget is so small, you let the district handle most of the 
_ formal activities. Are there informal," things that are done as part of 
your school level program?" 

"Then let 's < consider next the. district level. What are the evaluation 
activities that affect your school which the district carries out?" 

"Basically then, it's t}ie every other month visits and the test score 
reports?" - 

"What kind of effect dp. these* haVe on the program?" 

• ■ - / . 
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"So the personality thing makes a. big difference. Do you think that these 
district efforts could be made more useful to you?" 

How would you design the evaluation so would be more useful to the 
schools? 1 ! * j 

"Well, what information wou^d/ou like to have?" 

"What kind of support might they provide to make things easier for you?" 

"If the repotts were simpler, do you think people would actually read them? 

That is-^a- very interesting suggestion. Iidpft't think I've ever heard 
that idea before, and it makes so much sense," 

"Well, finally about the state level evaluation.* What about the PQR?" 

"Whoa,, let me interrupt -for a second. I -know that these things are an 
administrative nightmare. I've heard a wide range of comments about 
their actual benefit— -everything' from 'totally useless' to 'though they 
wejre a burden, there were some important benefits. v How would you 
characterize the value of PQR to this school?" 

"^.et's separate the PQR itself front the Mock Review, poes that make sense? 
"Air~"right then, what about the mpck review? It comes first doesn't it?" 
"What effect did it have on you and the school?" 
OK, let's talk about negative .impressions and then the positive ones." 



."You really think that, beyond the mere inconvenience, this had a 
negative effect on the actual school program?" 

"OK, what about its other influences. Tell me about some of the things 
you gained from the mock review." 

"Good. What^about it made the information useless to you?" 
"OK. Anything else?" ^ ' * 

"Well' then, if you were in charge and could redesign the mock review, 
how would you change it to make it more useful?" 

"I 'll note your objections to the length- and such. If you could change 
things to make the "information and the interaction more useful, what would 
you change?" 

"I guess it doesn't "lrtSke too much' sense to talk about the mock review 
without also talking about the PQ&. So tell me about it. How useful 
to you is it?" 

"Well, exactly what, as best as^you can recall, are the consequences of 
the PQR?" 

"Well, does it affect the program in any direct way?" 
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"What about people's attitudes or opinions? What changes do you see as 
a result?" * 

."Which factors in the PQR interfere with its u'sef ulness>£he most?" 
"Which are its strong points?" 

"If you. could redesign the whc-le PQR process, what changes would you 
make to make it more useful to the school program? * 

"Anything else?" 

"Good. While this is fresh in your memory, I want to go back to the two 
situations we discussed earlier and ask you if you now recall any evalua- 
tion influences that you didn't recall earlier. In the case of the- aides 
did any of these various evaluation factors enter into that decision?" 

"And in the case of the math program, PRISM, tell me about any* other 
evaluation influences you didn't recall when we^fp>rst talked." 

"Fine. I doubted that you'd missed anything, but I wanted to double 
check. The subtle factors are the ones we forget the most easily." 

4 

"I'm just about through with my questions, but before we conclude are 
there any additional comments you want to make about the things- we 've 
discussed? Is there anything else you want to add about the two program 
situation or about the different factors you felt contributed to those 
decisions, or about evaluation in other areas?" 

* * 

"In that case I want to thank you very much for your cooperation. You 
have been extremely helpful." 



r 
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Evaluation-User Survey 
Interview Suitpary Form 

Name of Interviewer: . School: 



Respondent: 
Title: 



Step I. After completing the interview, but before 'listening to the 
recording : 3 

-i.- Based on the complete "interview, describe in one paragraph the 
special ly-funded programs operating at this school. 

" ' — ; A- 



2. In one paragraph, describe the respondent's duties and responsibilities^ 
particularly as they involve the special programs you discussed. \ 
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3. In one paragraph each , describe the significant occurrences identified by the respondent 
and discussed in the interview. , 
Situation 1: 

r , 



f 



Situation 2: 
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4. For each occurrence: A. List (in approximate order of importance) the factors that influ- 
enced the final outcome. 

B. Summarize in one paragraph the interrelationships among these 
factors . 



1 



Situation 1: 



Situation 2: 
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5. Was evaluation information a factor in each of these situations? For each occurrence 
summarize in one paragraph the role of evaluation. \ 

Situation 1: \ • 



Situation 2: 



/ 
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6. In one paragraph each describe the respondents experiences with evaluation at the 
"school", "district", and "stat^ 1 levels. Indicate a), type of evaluation,^) its influence/ 
usefulness, c) factors contributing to its influence, d) ways of improving evaluation. 

"school level" activitities: 



'district level" activities: 
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PQR & mock review: 



7. Summarize in one paragraph any additional comments that were important. 
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8. In one paragraph describ^the^Bterview context—the salient features of {he setting, 
the participants and the interaction. 



\ 

\ 
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Step U: 




ning of each 



Replay the interview tape. (Set the counter at zero 000 %t 
new side.) f . . . * - 

As you listen to the interview:, / ' % 

1. Make additions/correction^ to the descriptive paragraphs you wrote in Step I 

2. Select important-quotes to illustrate key features, of the^ interview. * • 
0 Write out the quotes on the following pages. 

a. First indicate in a sentence or two. what is being discussed immediately / 
prior to the quote, i.e., son*? context for the' remark. If it is an answer 
to a particular question, give the question. ' . • 

b. Write the quote as accurately as you can. 

c. Don't forget, to indicate tire f tape counter reading at th# beginning and 
♦end-of each quote. * . * 



3. 



Key quotes : 

Topic Area — 

Context/Question : 

' 9 



T^pVCounter at beginning of quote 



Side A B ? 



Quote: 



4 



\ 



'Tape Counter at end pf 4 quote ■ Side A B ? 



TopicjArea . Tape Counter at beginning of quote ._ Side A B ?' 

/ 



Context/Question : 

V 



Quote: 



Topic Area 




Con\e wQuestion : 



Tape Counter at end of quote » Side A B ? 



Tape Counter at beginning of quote 



Side A B ? 



Quote: 



Tape Counter at end of quote 



Side A B •?' 
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Topic Area Tape^Counter at beginning of quote ' \ Side A ? 

Context/Question: _^ \ 

! \ 

\ 

: : " ' - \ 



Quote: 




T 



'' ^ Tape Counter at end of quote Side A B ? 

•■ ( 

♦ * 

Topic Area Tape Counter at beginning of quote Side A B ? 



Context/Question": 



.diL 



Quote: 



Tape Counter at end of quote- Side A B ? 



F-ll 



Topic Area Tape Counter at beginning of quote Side A B ? 

Context/Question: > ' ' 



Quote: 



Tape Counter at end of quote __ Side A B ? 

" - - ^> 

Topic Area . . ; ^ Tape Counter at t beginning of^ote ^ Side A B ? 

Context/Question: ' , 



Quote: 




^Tape Counter at .end of quote ^ ; Side A B ? 
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